Run Tests¶

Running Tests¶

ducktape discovers and runs tests in the path(s) provided. You can specify a folder with tests (all tests in Python modules named with “test_” prefix or “_test” suffix will be run), a specific test file (with any name) or even a specific class or test method, via absolute or relative paths. You can optionally specify a specific set of parameters for tests with @parametrize or @matrix annotations:

ducktape <relative_path_to_testdirectory>                   # e.g. ducktape dir/tests
ducktape <relative_path_to_file>                            # e.g. ducktape dir/tests/my_test.py
ducktape <path_to_test>[::SomeTestClass]                    # e.g. ducktape dir/tests/my_test.py::TestA
ducktape <path_to_test>[::SomeTestClass[.test_method]]      # e.g. ducktape dir/tests/my_test.py::TestA.test_a
ducktape <path_to_test>[::TestClass[.method[@params_json]]] # e.g. ducktape 'dir/tests/my_test.py::TestA.test_a@{"x": 100}'

Excluding Tests¶

Pass --exclude flag to exclude certain test(s) from the run, using the same syntax:

ducktape ./my_tests_dir --exclude ./my_tests_dir/test_a.py ./my_tests_dir/test_b.py::TestB.test_b

Test Suites¶

Test suite is a collection of tests to run, optionally also specifying which tests to exclude. Test suites are specified via YAML file

# list all tests that are part of the suite under the test suite name:
my_test_suite:
    - ./my_tests_dir/  # paths are relative to the test suite file location
    - ./another_tests_dir/test_file.py::TestClass.test_method  # same syntax as passing tests directly to ducktape
    - './another_tests_dir/test_file.py::TestClass.parametrized_method@{"x": 100}'  # params are supported too
    - ./third_tests_dir/prefix_*.py  # basic globs are supported (* and ? characters)

# each YAML file can contain one or more test suites:
another_test_suite:
    # you can optionally specify excluded tests in the suite as well using the following syntax:
    included:
        - ./some_tests_dir/
    excluded:
        - ./some_tests_dir/*_large_test.py

Running Test Suites¶

Tests suites are run in the same fashion as separate tests.

Run a single test suite:

ducktape ./path/to/test_suite.yml

Run multiple test suites:

ducktape ./path/to/test_suite_1.yml ./test_suite_2.yml

You can specify both tests and test suites at the same time:

ducktape ./my_test.py ./my_test_suite.yml ./another_test.py::TestClass.test_method

If the same test method is effectively specified more than once, it will only be executed once.

For example, if test_suite.yml lists test_a.py then running the following command will execute test_a.py only once:

ducktape test_suite.yml test_a.py

If you specify a folder, all tests (ie python files) under that folder will be discovered, but test suites will be not.

For example, if test_dir contains my_test.py and my_test_suite.yml, then running:

ducktape ./test_dir

will execute my_test.py but skip my_test_suite.yml.

To execute both my_test.py and my_test_suite.yml you need to specify test suite path explicitly:

ducktape ./test_dir/ ./test_dir/my_test_suite.yml

Exclude and Test Suites¶

Exclude section in the test suite applies only to that test suite. --exclude parameter passed to ducktape applies to all loaded tests and test suites.

For example, if test_dir contains test_a.py, test_b.py and test_c.py, and test_suite.yml is:

suite_one:
    included:
        - ./test_dir/*.py
    excluded:
        - ./test_dir/test_a.py
suite_two:
    included:
        - ./test_dir/
    excluded:
        - ./test_dir/test_b.py

Then running:

ducktape test_suite.yml

runs each of test_a.py, test_b.py and test_c.py once

But running:

ducktape test_suite.yml --exclude test_dir/test_a.py

runs only test_b.py and test_c.py once, and skips test_a.py.

Options¶

To see a complete listing of options run:

ducktape --help

Discover and run your tests

usage: ducktape [-h] [--exclude [EXCLUDE [EXCLUDE ...]]] [--collect-only]
                [--collect-num-nodes] [--debug] [--config-file CONFIG_FILE]
                [--compress] [--cluster CLUSTER]
                [--default-num-nodes DEFAULT_NUM_NODES]
                [--cluster-file CLUSTER_FILE] [--results-root RESULTS_ROOT]
                [--exit-first] [--no-teardown] [--version]
                [--parameters PARAMETERS] [--globals GLOBALS]
                [--max-parallel MAX_PARALLEL] [--repeat REPEAT]
                [--subsets SUBSETS] [--subset SUBSET]
                [--historical-report HISTORICAL_REPORT]
                [--skip-nodes-allocation] [--sample SAMPLE]
                [--fail-bad-cluster-utilization] [--fail-greedy-tests]
                [--test-runner-timeout TEST_RUNNER_TIMEOUT]
                [--ssh-checker-function SSH_CHECKER_FUNCTION [SSH_CHECKER_FUNCTION ...]]
                [--deflake DEFLAKE]
                [test_path [test_path ...]]

Required Arguments

`test_path=['/home/docs/checkouts/readthedocs.org/user_builds/ducktape/checkouts/latest/docs']`
	One or more test identifiers or test suite paths to execute

Optional Arguments

`--exclude`	one or more space-delimited strings indicating which tests to exclude
`--collect-only=False`
	display collected tests, but do not run.
`--collect-num-nodes=False`
	display total number of nodes requested by all tests, but do not run anything.
`--debug=False`	pipe more verbose test output to stdout.
`--config-file="~/.ducktape/config"`
	path to project-specific configuration file.
`--compress=False`
	compress remote logs before collection.
`--cluster="ducktape.cluster.vagrant.VagrantCluster"`
	cluster class to use to allocate nodes for tests.
`--default-num-nodes`
	Global hint for cluster usage. A test without the @cluster annotation will default to this value for expected cluster usage.
`--cluster-file`	path to a json file which provides information needed to initialize a json cluster. The file is used to read/write cached cluster info if cluster is ducktape.cluster.vagrant.VagrantCluster.
`--results-root="./results"`
	path to custom root results directory. Running ducktape with this root specified will result in new test results being stored in a subdirectory of this root directory.
`--exit-first=False`
	exit after first failure
`--no-teardown=False`
	don’t kill running processes or remove log files when a test has finished running. This is primarily useful for test developers who want to interact with running services after a test has run.
`--version=False`
	display version
`--parameters`	inject these arguments into the specified test(s). Specify parameters as a JSON string.
`--globals`	user-defined globals go here. This can be a file containing a JSON object, or a string representing a JSON object.
`--max-parallel=1`
	Upper bound on number of tests run simultaneously.
`--repeat=1`	Use this flag to repeat all discovered tests the given number of times.
`--subsets=1`	Number of subsets of tests to statically break the tests into to allow for parallel execution without coordination between test runner processes.
`--subset=0`	Which subset of the tests to run, based on the breakdown using the parameter for –subsets
`--historical-report`
	URL of a JSON report file containing stats from a previous test run. If specified, this will be used when creating subsets of tests to divide evenly by total run time instead of by number of tests.
`--skip-nodes-allocation=False`
	Use this flag to skip allocating nodes for services. Can be used when running specific tests on a running platform
`--sample`	The size of a random test sample to run
`--fail-bad-cluster-utilization=False`
	Fail a test if the test declared that it needs more nodes than it actually used. E.g. if the test had `@cluster(num_nodes=10)` annotation, but never used more than 5 nodes during its execution.
`--fail-greedy-tests=False`
	Fail a test if it has no @cluster annotation or if @cluster annotation is empty. You can still specify 0-sized cluster explicitly using either num_nodes=0 or cluster_spec=ClusterSpec.empty()
`--test-runner-timeout=1800000`
	Amount of time in milliseconds between test communicating between the test runner before a timeout error occurs. Default is 30 minutes
`--ssh-checker-function`
	Python module path(s) to a function that takes an exception and a remote account that will be called when an ssh error occurs, this can give some validation or better logging when an ssh error occurs. Specify any number of module paths after this flag to be called.
`--deflake=1`	the number of times a failed test should be ran in total (including its initial run) to determine flakyness. When not present, deflake will not be used, and a test will be marked as either passed or failed. When enabled tests will be marked as flaky if it passes on any of the reruns

Configuration File¶

You can configure options in three locations: on the command line (highest priority), in a user configuration file in ~/.ducktape/config, and in a project-specific configuration <project_dir>/.ducktape/config (lowest priority). Configuration files use the same syntax as command line arguments and may split arguments across multiple lines:

--debug
--exit-first
--cluster=ducktape.cluster.json.JsonCluster

Output¶

Test results go in results/<session_id>.<session_id> which looks like <date>--<test_number>. For example: results/2015-03-28--002

ducktape does its best to group test results and log files in a sensible way. The output directory is structured like so:

<session_id>
    session_log.info
    session_log.debug
    report.txt   # Summary report of all tests run in this session
    report.html  # Open this to see summary report in a browser
    report.css

    <test_class_name>
        <test_method_name>
            test_log.info
            test_log.debug
            report.txt   # Report on this single test
            [data.json]  # Present if the test returns data

            <service_1>
                <node_1>
                    some_logs
                <node_2>
                    some_logs
    ...

To see an example of the output structure, go here and click on one of the details links.