Rerunning tests#
By default, only tests that had previously not run will be rerun, unless the test is explicitly requested via keyowrd or other filters.
Filter tests based on previous status#
In rerun mode, the previous test status is included implicitly as a test keyword which allows filtering based on previous statuses.
Examples#
$ canary run ./status
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from status
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 5 test specs from 5 generators
INFO: Caching test specs
INFO: Created selection 'scarlet-spur'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-45-29.264365
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────
fail de70161 SUBMITTED 1/5
fail de70161 STARTED 1/5
fail de70161 FAIL (FAILED) 0.6s 1/5
diff 8e033df SUBMITTED 2/5
diff 8e033df STARTED 2/5
diff 8e033df FAIL (DIFFED) 0.3s 2/5
skip e0e106b SUBMITTED 3/5
skip e0e106b STARTED 3/5
skip e0e106b SKIP (SKIPPED) 0.3s 3/5
pass e5e852a SUBMITTED 4/5
pass e5e852a STARTED 4/5
pass e5e852a PASS (SUCCESS) 0.3s 4/5
timeout c11972b SUBMITTED 5/5
timeout c11972b STARTED 5/5
timeout c11972b FAIL (TIMEOUT) 2.2s 5/5
┌─────────┬─────────┬────────────────┬─────────┬──────────────────────────────────────┐
│ Job │ ID │ Status │ Elapsed │ Details │
├─────────┼─────────┼────────────────┼─────────┼──────────────────────────────────────┤
│ fail │ de70161 │ FAIL (FAILED) │ 0.3s │ Test exited with exit code = 65 │
│ diff │ 8e033df │ FAIL (DIFFED) │ 0.3s │ Test exited with diff exit code = 64 │
│ skip │ e0e106b │ SKIP (SKIPPED) │ 0.3s │ Test exited with skip exit code = 80 │
│ timeout │ c11972b │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
└─────────┴─────────┴────────────────┴─────────┴──────────────────────────────────────┘
5/5 COMPLETE, 1 SUCCESS, 1 SKIPPED, 1 DIFFED, 1 FAILED, 1 TIMEOUT, in 00:00:03
INFO: Finished session in 3.95 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
Rerun all failed tests#
$ canary run -k 'not success'
INFO: Running tests in tag :all:
INFO: Selecting specs based on 1 rules
INFO: Selecting test jobs based on runtime environment
INFO: Excluded 1 test jobs
Reason Count
───────────────────────────────────
previous result = SUCCESS 1
INFO: Starting session 2026-06-04T20-45-33.851681
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────
diff 8e033df SUBMITTED 1/4
diff 8e033df STARTED 1/4
diff 8e033df FAIL (DIFFED) 0.6s 1/4
fail de70161 SUBMITTED 2/4
fail de70161 STARTED 2/4
fail de70161 FAIL (FAILED) 0.3s 2/4
skip e0e106b SUBMITTED 3/4
skip e0e106b STARTED 3/4
skip e0e106b SKIP (SKIPPED) 0.3s 3/4
timeout c11972b SUBMITTED 4/4
timeout c11972b STARTED 4/4
timeout c11972b FAIL (TIMEOUT) 2.2s 4/4
┌─────────┬─────────┬────────────────┬─────────┬──────────────────────────────────────┐
│ Job │ ID │ Status │ Elapsed │ Details │
├─────────┼─────────┼────────────────┼─────────┼──────────────────────────────────────┤
│ diff │ 8e033df │ FAIL (DIFFED) │ 0.3s │ Test exited with diff exit code = 64 │
│ fail │ de70161 │ FAIL (FAILED) │ 0.3s │ Test exited with exit code = 65 │
│ skip │ e0e106b │ SKIP (SKIPPED) │ 0.3s │ Test exited with skip exit code = 80 │
│ timeout │ c11972b │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
└─────────┴─────────┴────────────────┴─────────┴──────────────────────────────────────┘
4/4 COMPLETE, 1 SKIPPED, 1 DIFFED, 1 FAILED, 1 TIMEOUT, in 00:00:03
INFO: Finished session in 3.60 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
Rerun only the diffed tests#
$ canary run -k diff
INFO: Running tests in tag :all:
INFO: Selecting specs based on 1 rules
INFO: Selected 1 test specs
INFO: Excluded 4 test specs during selection
Reason Count
─────────────────────────────────────────────────
keyword expression 'diff' did not match 4
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-45-38.094793
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────
diff 8e033df SUBMITTED 1/1
diff 8e033df STARTED 1/1
diff 8e033df FAIL (DIFFED) 0.6s 1/1
┌──────┬─────────┬───────────────┬─────────┬──────────────────────────────────────┐
│ Job │ ID │ Status │ Elapsed │ Details │
├──────┼─────────┼───────────────┼─────────┼──────────────────────────────────────┤
│ diff │ 8e033df │ FAIL (DIFFED) │ 0.3s │ Test exited with diff exit code = 64 │
└──────┴─────────┴───────────────┴─────────┴──────────────────────────────────────┘
1/1 COMPLETE, 1 DIFFED, in 00:00:00.65
INFO: Finished session in 0.71 s. with returncode 2
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
Rerun tests inside the view#
Optionally, a subdirectory of the workspace view argument can be passed to canary run, causing canary to rerun only those tests that are in PATH and its children:
$ canary run ./TestResults/pass
INFO: Running tests from view paths
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-45-39.422506
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
─────────────────────────────────────────────────────────────────────────────────
pass e5e852a SUBMITTED 1/1
pass e5e852a STARTED 1/1
pass e5e852a PASS (SUCCESS) 0.6s 1/1
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.70 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults