Rerunning tests#
By default, only tests that had previously not run will be rerun, unless the test is explicitly requested via keyowrd or other filters.
Filter tests based on previous status#
In rerun mode, the previous test status is included implicitly as a test keyword which allows filtering based on previous statuses.
Examples#
$ canary run ./status
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from status
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 5 test specs from 5 generators
INFO: Caching test specs
INFO: Created selection 'agile-autumn'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-06.709529
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────
pass 2d8ffb8 SUBMITTED 2/5
pass 2d8ffb8 STARTED 0.3s 2/5
diff b4597e3 SUBMITTED 1/5
diff b4597e3 STARTED 0.3s 1/5
pass 2d8ffb8 PASS (SUCCESS) 0.2s 0.5s 2/5
diff b4597e3 FAIL (DIFFED) 0.3s 0.5s 1/5
fail a850e81 SUBMITTED 3/5
fail a850e81 STARTED 0.0s 3/5
skip e01f382 SUBMITTED 4/5
skip e01f382 STARTED 0.0s 4/5
fail a850e81 FAIL (FAILED) 0.0s 0.2s 3/5
skip e01f382 SKIP (SKIPPED) 0.0s 0.2s 4/5
timeout 7127eb6 SUBMITTED 5/5
timeout 7127eb6 STARTED 0.0s 5/5
timeout 7127eb6 FAIL (TIMEOUT) 0.0s 2.2s 5/5
┌────────────┬─────────────┬──────────────────┬─────────────┬────────────┬─────────────────────────────────────────────┐
│ Job │ ID │ Status │ Queued │ Elapsed │ Details │
├────────────┼─────────────┼──────────────────┼─────────────┼────────────┼─────────────────────────────────────────────┤
│ diff │ b4597e3 │ FAIL (DIFFED) │ 0.0s │ 0.2s │ Test exited with diff exit code = 64 │
│ fail │ a850e81 │ FAIL (FAILED) │ 0.0s │ 0.2s │ Test exited with exit code = 65 │
│ skip │ e01f382 │ SKIP (SKIPPED) │ 0.0s │ 0.2s │ Test exited with skip exit code = 80 │
│ timeout │ 7127eb6 │ FAIL (TIMEOUT) │ 0.0s │ 2.2s │ Job timed out after 2.0 s. │
└────────────┴─────────────┴──────────────────┴─────────────┴────────────┴─────────────────────────────────────────────┘
5/5 COMPLETE, 1 SUCCESS, 1 SKIPPED, 1 DIFFED, 1 FAILED, 1 TIMEOUT, in 00:00:02
INFO: Finished session in 2.97 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Rerun all failed tests#
$ canary run -k 'not success'
INFO: Running tests in tag :all:
INFO: Selecting specs based on 1 rules
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-10.222354
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────
diff b4597e3 SUBMITTED 2/4
diff b4597e3 STARTED 0.3s 2/4
fail a850e81 SUBMITTED 1/4
fail a850e81 STARTED 0.3s 1/4
diff b4597e3 FAIL (DIFFED) 0.2s 0.5s 2/4
fail a850e81 FAIL (FAILED) 0.3s 0.5s 1/4
skip e01f382 SUBMITTED 3/4
skip e01f382 STARTED 0.0s 3/4
timeout 7127eb6 SUBMITTED 4/4
timeout 7127eb6 STARTED 0.0s 4/4
skip e01f382 SKIP (SKIPPED) 0.0s 0.2s 3/4
timeout 7127eb6 FAIL (TIMEOUT) 0.0s 2.2s 4/4
┌────────────┬─────────────┬──────────────────┬─────────────┬────────────┬─────────────────────────────────────────────┐
│ Job │ ID │ Status │ Queued │ Elapsed │ Details │
├────────────┼─────────────┼──────────────────┼─────────────┼────────────┼─────────────────────────────────────────────┤
│ diff │ b4597e3 │ FAIL (DIFFED) │ 0.0s │ 0.2s │ Test exited with diff exit code = 64 │
│ fail │ a850e81 │ FAIL (FAILED) │ 0.0s │ 0.2s │ Test exited with exit code = 65 │
│ skip │ e01f382 │ SKIP (SKIPPED) │ 0.0s │ 0.2s │ Test exited with skip exit code = 80 │
│ timeout │ 7127eb6 │ FAIL (TIMEOUT) │ 0.0s │ 2.2s │ Job timed out after 2.0 s. │
└────────────┴─────────────┴──────────────────┴─────────────┴────────────┴─────────────────────────────────────────────┘
4/4 COMPLETE, 1 SKIPPED, 1 DIFFED, 1 FAILED, 1 TIMEOUT, in 00:00:02
INFO: Finished session in 2.72 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Rerun only the diffed tests#
$ canary run -k diff
INFO: Running tests in tag :all:
INFO: Selecting specs based on 1 rules
INFO: Selected 1 test specs
INFO: Excluded 3 test specs during selection
Reason Count
─────────────────────────────────────────────────
keyword expression 'diff' did not match 3
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-13.473802
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────────────
diff b4597e3 SUBMITTED 1/1
diff b4597e3 STARTED 0.3s 1/1
diff b4597e3 FAIL (DIFFED) 0.3s 0.5s 1/1
┌────────┬─────────────┬──────────────────┬─────────────┬─────────────┬────────────────────────────────────────────────┐
│ Job │ ID │ Status │ Queued │ Elapsed │ Details │
├────────┼─────────────┼──────────────────┼─────────────┼─────────────┼────────────────────────────────────────────────┤
│ diff │ b4597e3 │ FAIL (DIFFED) │ 0.0s │ 0.2s │ Test exited with diff exit code = 64 │
└────────┴─────────────┴──────────────────┴─────────────┴─────────────┴────────────────────────────────────────────────┘
1/1 COMPLETE, 1 DIFFED, in 00:00:00.47
INFO: Finished session in 0.53 s. with returncode 2
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Rerun tests inside the view#
Optionally, a subdirectory of the workspace view argument can be passed to canary run, causing canary to rerun only those tests that are in PATH and its children:
$ canary run ./TestResults/pass
INFO: Running tests from view paths
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-14.513842
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
───────────────────────────────────────────────────────────────────────────────────────────
pass 2d8ffb8 SUBMITTED 1/1
pass 2d8ffb8 STARTED 0.3s 1/1
pass 2d8ffb8 PASS (SUCCESS) 0.2s 0.5s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.54 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults