Running tests#
Use canary run to run tests.
Basic usage#
$ canary run ./basic
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'deft-owl'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-15.804590
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────────────────
first 260b4b0 SUBMITTED 1/2
first 260b4b0 STARTED 0.3s 1/2
second 57a9775 SUBMITTED 2/2
second 57a9775 STARTED 0.3s 2/2
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/2
second 57a9775 PASS (SUCCESS) 0.3s 0.6s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 0.65 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Filter tests to run by keyword#
canary run -k KEYWORD_EXPR PATH [PATHS...]
where KEYWORD_EXPR is a Python expression such as -k 'fast and regression'. For example
$ canary run -k first ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Selecting specs based on 1 rules
INFO: Selected 1 test specs
INFO: Excluded 1 test spec during selection
Reason Count
──────────────────────────────────────────────────
keyword expression 'first' did not match 1
INFO: Created selection 'steel-sky'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-17.174554
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────
first 260b4b0 SUBMITTED 1/1
first 260b4b0 STARTED 0.3s 1/1
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.54 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Limit the number of concurrent tests#
canary run --workers=N PATH [PATHS...]
where N is a number of workers. For example,
$ canary run --workers=1 ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'cornflower-moraine'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-18.426216
INFO: Starting process pool with max 1 workers
Job ID Status Queued Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────────────────
first 260b4b0 SUBMITTED 1/2
first 260b4b0 STARTED 0.3s 1/2
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/2
second 57a9775 SUBMITTED 2/2
second 57a9775 STARTED 0.0s 2/2
second 57a9775 PASS (SUCCESS) 0.0s 0.3s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 0.86 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Set a timeout on the test session#
canary run --timeout session=T PATH [PATHS...]
where T is a duration in Go’s duration format (40s,, 1h20m, 2h, 4h30m30s, etc.) For example,
$ canary run --timeout session=1m ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'beige-reef'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-20.040075
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────────────────
second 57a9775 SUBMITTED 2/2
second 57a9775 STARTED 0.3s 2/2
first 260b4b0 SUBMITTED 1/2
first 260b4b0 STARTED 0.3s 1/2
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/2
second 57a9775 PASS (SUCCESS) 0.3s 0.6s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 0.64 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Run specific test files#
Run a file directly#
Test files can be run directly by passing their paths to canary run
$ canary run ./basic/first/first.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic/first
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'ink-galaxy'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-21.409670
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────
first 260b4b0 SUBMITTED 1/1
first 260b4b0 STARTED 0.3s 1/1
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.55 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
$ ls -F TestResults
VIEW.TAG
first@
If a path separator is replaced with a colon :, the path is interpreted as root:path. ie, path segments after the : are used as the relative path to the test execution directory:
$ canary run .:basic/first/first.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'firebrick-wren'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-22.674615
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────
first 260b4b0 SUBMITTED 1/1
first 260b4b0 STARTED 0.3s 1/1
first 260b4b0 PASS (SUCCESS) 0.3s 0.5s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.54 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
$ ls -F TestResults
VIEW.TAG
basic/
Running tests from a file#
Select tests can be executed by specifying their paths in a json or yaml configuration file with the following layout:
testpaths:
- root: <root>
paths:
- <path_1>
- <path_2>
...
- <path_n>
where <root> is a parent directory of the tests and <path_i> are the file paths relative to <root>. If <root> is a relative path, it is considered relative to the path of the configuration file. Consider, for example, the examples directory tree:
$ canary tree --exclude-results .
.
├── analyze_only
│ └── analyze_only.pyt
├── basic
│ ├── first
│ │ └── first.pyt
│ └── second
│ ├── add.py
│ └── second.pyt
├── canary.json
├── centered_space
│ └── centered_space.pyt
├── copy_and_link
│ ├── copy.txt
│ ├── copy_and_link.pyt
│ └── link.txt
├── ctest
│ ├── CTestTestfile.cmake
│ ├── resource_group_test_1.py
│ └── resource_group_test_2.py
├── depends_on
│ ├── basic
│ │ ├── a.pyt
│ │ └── depends_on_a.pyt
│ ├── multi
│ │ ├── blt.pyt
│ │ ├── green_eggs_and_ham.pyt
│ │ ├── ingredients.pyt
│ │ ├── kitchen_sink.py
│ │ └── kitchen_sink.pyt
│ ├── parameter
│ │ ├── breakfast.pyt
│ │ └── lunch.pyt
│ └── result
│ ├── depends_on_willfail.pyt
│ └── willfail.pyt
├── enable
│ └── enable.pyt
├── execute_and_analyze
│ └── execute_and_analyze.pyt
├── instance_attributes
│ └── instance_attributes.pyt
├── junit.xml
├── parameterize
│ ├── parameterize1.pyt
│ ├── parameterize2.pyt
│ ├── parameterize3.pyt
│ └── parameterize4.pyt
├── plugin
│ └── my_command.py
├── random_space
│ └── random_space.pyt
├── rebaseline
│ ├── rebaseline.pyt
│ └── test.base.out
├── resources
│ └── resources.pyt
├── status
│ ├── diff.pyt
│ ├── fail.pyt
│ ├── pass.pyt
│ ├── skip.pyt
│ └── timeout.pyt
├── tests.json
├── timeoutx
│ └── timeout.pyt
├── vvt
│ └── test_exec_dir.vvt
└── xstatus
├── xdiff-fail.pyt
├── xdiff.pyt
├── xfail-code.pyt
├── xfail-fail.pyt
└── xfail.pyt
To run only centered_space/centered_space.pyt and parameterize/parameterize2.pyt, write the following to tests.json
{
"testpaths": [
{
"root": ".",
"paths": ["centered_space/centered_space.pyt", "parameterize/parameterize2.pyt"]
}
]
}
and pass it to canary run:
$ canary run -f tests.json
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 12 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'aquamarine-grove'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-24.419808
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
centered_space.a=0.b=0 97f4942 SUBMITTED 2/12
centered_space.a=0.b=0 97f4942 STARTED 0.3s 2/12
parameterize2.a=1.b=2 5adc387 SUBMITTED 1/12
parameterize2.a=1.b=2 5adc387 STARTED 0.3s 1/12
centered_space.a=0.b=0 97f4942 PASS (SUCCESS) 0.2s 0.5s 2/12
parameterize2.a=1.b=2 5adc387 PASS (SUCCESS) 0.3s 0.5s 1/12
centered_space.a=0.b=2 4e1241e SUBMITTED 4/12
centered_space.a=0.b=2 4e1241e STARTED 0.0s 4/12
centered_space.a=10.b=0 62ecc7b SUBMITTED 3/12
centered_space.a=10.b=0 62ecc7b STARTED 0.0s 3/12
centered_space.a=0.b=2 4e1241e PASS (SUCCESS) 0.0s 0.2s 4/12
centered_space.a=10.b=0 62ecc7b PASS (SUCCESS) 0.0s 0.2s 3/12
centered_space.a=0.b=1 94c6460 SUBMITTED 5/12
centered_space.a=0.b=1 94c6460 STARTED 0.0s 5/12
centered_space.a=0.b=-1 2fc444b SUBMITTED 6/12
centered_space.a=0.b=-1 2fc444b STARTED 0.0s 6/12
centered_space.a=0.b=1 94c6460 PASS (SUCCESS) 0.0s 0.2s 5/12
centered_space.a=0.b=-1 2fc444b PASS (SUCCESS) 0.0s 0.2s 6/12
centered_space.a=5.b=0 bd5345d SUBMITTED 8/12
centered_space.a=5.b=0 bd5345d STARTED 0.0s 8/12
centered_space.a=0.b=-2 4c8ebb3 SUBMITTED 7/12
centered_space.a=0.b=-2 4c8ebb3 STARTED 0.0s 7/12
centered_space.a=5.b=0 bd5345d PASS (SUCCESS) 0.0s 0.2s 8/12
centered_space.a=0.b=-2 4c8ebb3 PASS (SUCCESS) 0.0s 0.2s 7/12
parameterize2.a=5.b=6 f78da16 SUBMITTED 10/12
parameterize2.a=5.b=6 f78da16 STARTED 0.0s 10/12
centered_space.a=-5.b=0 fb81aad SUBMITTED 9/12
centered_space.a=-5.b=0 fb81aad STARTED 0.0s 9/12
parameterize2.a=5.b=6 f78da16 PASS (SUCCESS) 0.0s 0.2s 10/12
centered_space.a=-5.b=0 fb81aad PASS (SUCCESS) 0.0s 0.2s 9/12
centered_space.a=-10.b=0 9e7c62d SUBMITTED 11/12
centered_space.a=-10.b=0 9e7c62d STARTED 0.0s 11/12
centered_space.a=-10.b=0 9e7c62d PASS (SUCCESS) 0.0s 0.2s 11/12
centered_space b1f9ac2 SUBMITTED 12/12
centered_space b1f9ac2 STARTED 0.0s 12/12
centered_space b1f9ac2 PASS (SUCCESS) 0.0s 0.2s 12/12
INFO: 12/12 tests finished with status PASS
INFO: Finished session in 1.99 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults