Running tests#
Use canary run to run tests.
Basic usage#
$ canary run ./basic
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'daring-cinder'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-53-54.801233
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/2
first 4b172fc STARTED 1/2
first 4b172fc PASS (SUCCESS) 0.6s 1/2
second b1f18b4 SUBMITTED 2/2
second b1f18b4 STARTED 2/2
second b1f18b4 PASS (SUCCESS) 0.3s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 1.03 s. with returncode 0
INFO: HTML report written to Canary.html
Filter tests to run by keyword#
canary run -k KEYWORD_EXPR PATH [PATHS...]
where KEYWORD_EXPR is a Python expression such as -k 'fast and regression'. For example
$ canary run -k first ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Selecting specs based on 1 rules
INFO: Selected 1 test specs
INFO: Excluded 1 test spec during selection
Reason Count
──────────────────────────────────────────────────
keyword expression 'first' did not match 1
INFO: Created selection 'chestnut-beacon'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-53-56.922598
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/1
first 4b172fc STARTED 1/1
first 4b172fc PASS (SUCCESS) 0.6s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.68 s. with returncode 0
INFO: HTML report written to Canary.html
Limit the number of concurrent tests#
canary run --workers=N PATH [PATHS...]
where N is a number of workers. For example,
$ canary run --workers=1 ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'certain-talon'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-53-58.674265
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/2
first 4b172fc STARTED 1/2
first 4b172fc PASS (SUCCESS) 0.6s 1/2
second b1f18b4 SUBMITTED 2/2
second b1f18b4 STARTED 2/2
second b1f18b4 PASS (SUCCESS) 0.3s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 1.03 s. with returncode 0
INFO: HTML report written to Canary.html
Set a timeout on the test session#
canary run --timeout session=T PATH [PATHS...]
where T is a duration in Go’s duration format (40s,, 1h20m, 2h, 4h30m30s, etc.) For example,
$ canary run --timeout session=1m ./basic
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 2 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'viridian-cinder'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-54-00.766273
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/2
first 4b172fc STARTED 1/2
first 4b172fc PASS (SUCCESS) 0.6s 1/2
second b1f18b4 SUBMITTED 2/2
second b1f18b4 STARTED 2/2
second b1f18b4 PASS (SUCCESS) 0.3s 2/2
INFO: 2/2 tests finished with status PASS
INFO: Finished session in 1.03 s. with returncode 0
INFO: HTML report written to Canary.html
Run specific test files#
Run a file directly#
Test files can be run directly by passing their paths to canary run
$ canary run ./basic/first/first.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic/first
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'loyal-fern'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-54-02.750483
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/1
first 4b172fc STARTED 1/1
first 4b172fc PASS (SUCCESS) 0.6s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.68 s. with returncode 0
INFO: HTML report written to Canary.html
$ ls -F TestResults
first@
If a path separator is replaced with a colon :, the path is interpreted as root:path. ie, path segments after the : are used as the relative path to the test execution directory:
$ canary run .:basic/first/first.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'winter-flint'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-54-04.384123
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────
first 4b172fc SUBMITTED 1/1
first 4b172fc STARTED 1/1
first 4b172fc PASS (SUCCESS) 0.6s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.68 s. with returncode 0
INFO: HTML report written to Canary.html
$ ls -F TestResults
basic/
Running tests from a file#
Select tests can be executed by specifying their paths in a json or yaml configuration file with the following layout:
testpaths:
- root: <root>
paths:
- <path_1>
- <path_2>
...
- <path_n>
where <root> is a parent directory of the tests and <path_i> are the file paths relative to <root>. If <root> is a relative path, it is considered relative to the path of the configuration file. Consider, for example, the examples directory tree:
$ canary tree --exclude-results .
.
├── Canary.html@
├── HTML
│ ├── Pass.html
│ ├── Total.html
│ ├── index.html
│ ├── jobs
│ │ ├── 4b172fcef2eb368584aeec8abc4f7409dda7c131d4bd5a1d99d79af9726619cc.html
│ │ └── b1f18b409e6d4e06a34c9601f7ba4dbb3f96c16647aa20bdc736dfd9378f678d.html
│ └── manifest.json
├── MARKDOWN
│ ├── 4b172fcef2eb368584aeec8abc4f7409dda7c131d4bd5a1d99d79af9726619cc.md
│ ├── Pass.md
│ ├── Total.md
│ ├── b1f18b409e6d4e06a34c9601f7ba4dbb3f96c16647aa20bdc736dfd9378f678d.md
│ ├── index.md
│ └── manifest.json
├── analyze_only
│ └── analyze_only.pyt
├── artifact.pyt
├── attributes.pyt
├── basic
│ ├── first
│ │ ├── Canary.html@
│ │ └── first.pyt
│ └── second
│ ├── add.py
│ └── second.pyt
├── canary.json
├── centered_space
│ └── centered_space.pyt
├── copy_and_link
│ ├── copy.txt
│ ├── copy_and_link.1.txt
│ ├── copy_and_link.2.txt
│ ├── copy_and_link.pyt
│ └── link.txt
├── ctest
│ ├── CTestTestfile.cmake
│ ├── resource_group_test_1.py
│ └── resource_group_test_2.py
├── depends_on
│ ├── basic
│ │ ├── a.pyt
│ │ └── depends_on_a.pyt
│ ├── multi
│ │ ├── blt.pyt
│ │ ├── green_eggs_and_ham.pyt
│ │ ├── ingredients.pyt
│ │ ├── kitchen_sink.py
│ │ └── kitchen_sink.pyt
│ ├── parameter
│ │ ├── breakfast.pyt
│ │ └── lunch.pyt
│ └── result
│ ├── depends_on_willfail.pyt
│ └── willfail.pyt
├── enable
│ └── enable.pyt
├── exclusive.pyt
├── execute_and_analyze
│ └── execute_and_analyze.pyt
├── instance_attributes
│ └── instance_attributes.pyt
├── junit.xml
├── owners.pyt
├── parameterize
│ ├── parameterize1.pyt
│ ├── parameterize2.pyt
│ ├── parameterize3.pyt
│ └── parameterize4.pyt
├── plugin
│ └── my_command.py
├── random_space
│ └── random_space.pyt
├── rebaseline
│ ├── rebaseline.pyt
│ └── test.base.out
├── resources
│ ├── cpus.pyt
│ └── resources.pyt
├── skipif.pyt
├── status
│ ├── diff.pyt
│ ├── fail.pyt
│ ├── pass.pyt
│ ├── skip.pyt
│ └── timeout.pyt
├── testname.pyt
├── tests.json
├── timeoutx
│ └── timeout.pyt
├── vvt
│ ├── common_directives.vvtinc
│ ├── include.vvt
│ └── test_exec_dir.vvt
└── xstatus
├── xdiff-fail.pyt
├── xdiff.pyt
├── xfail-code.pyt
├── xfail-fail.pyt
└── xfail.pyt
To run only centered_space/centered_space.pyt and parameterize/parameterize2.pyt, write the following to tests.json
{
"testpaths": [
{
"root": ".",
"paths": ["centered_space/centered_space.pyt", "parameterize/parameterize2.pyt"]
}
]
}
and pass it to canary run:
$ canary run -f tests.json
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 12 test specs from 2 generators
INFO: Caching test specs
INFO: Created selection 'plum-cinder'
INFO: Selecting test jobs based on runtime environment
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-54-06.828773
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────────────────────────
parameterize2.a=1.b=2 b741505 SUBMITTED 1/12
parameterize2.a=1.b=2 b741505 STARTED 1/12
parameterize2.a=1.b=2 b741505 PASS (SUCCESS) 0.7s 1/12
centered_space.a=0.b=0 18d1ae9 SUBMITTED 2/12
centered_space.a=0.b=0 18d1ae9 STARTED 2/12
centered_space.a=0.b=0 18d1ae9 PASS (SUCCESS) 0.4s 2/12
centered_space.a=10.b=0 6fbb430 SUBMITTED 3/12
centered_space.a=10.b=0 6fbb430 STARTED 3/12
centered_space.a=10.b=0 6fbb430 PASS (SUCCESS) 0.4s 3/12
centered_space.a=0.b=2 6c9223c SUBMITTED 4/12
centered_space.a=0.b=2 6c9223c STARTED 4/12
centered_space.a=0.b=2 6c9223c PASS (SUCCESS) 0.4s 4/12
centered_space.a=0.b=1 f237a83 SUBMITTED 5/12
centered_space.a=0.b=1 f237a83 STARTED 5/12
centered_space.a=0.b=1 f237a83 PASS (SUCCESS) 0.4s 5/12
centered_space.a=0.b=-1 4947e35 SUBMITTED 6/12
centered_space.a=0.b=-1 4947e35 STARTED 6/12
centered_space.a=0.b=-1 4947e35 PASS (SUCCESS) 0.4s 6/12
centered_space.a=0.b=-2 071c9c3 SUBMITTED 7/12
centered_space.a=0.b=-2 071c9c3 STARTED 7/12
centered_space.a=0.b=-2 071c9c3 PASS (SUCCESS) 0.4s 7/12
centered_space.a=5.b=0 d8c3599 SUBMITTED 8/12
centered_space.a=5.b=0 d8c3599 STARTED 8/12
centered_space.a=5.b=0 d8c3599 PASS (SUCCESS) 0.4s 8/12
centered_space.a=-5.b=0 c300f2d SUBMITTED 9/12
centered_space.a=-5.b=0 c300f2d STARTED 9/12
centered_space.a=-5.b=0 c300f2d PASS (SUCCESS) 0.4s 9/12
parameterize2.a=5.b=6 95fd37e SUBMITTED 10/12
parameterize2.a=5.b=6 95fd37e STARTED 10/12
parameterize2.a=5.b=6 95fd37e PASS (SUCCESS) 0.4s 10/12
centered_space.a=-10.b=0 9a6e478 SUBMITTED 11/12
centered_space.a=-10.b=0 9a6e478 STARTED 11/12
centered_space.a=-10.b=0 9a6e478 PASS (SUCCESS) 0.4s 11/12
centered_space 97d51af SUBMITTED 12/12
centered_space 97d51af STARTED 12/12
centered_space 97d51af PASS (SUCCESS) 0.5s 12/12
INFO: 12/12 tests finished with status PASS
INFO: Finished session in 5.87 s. with returncode 0
INFO: HTML report written to Canary.html