Running tests in a batch scheduler#
Execution in a batch scheduler is accomplished by defining the scheduler and batching scheme on the command line:
canary run -b scheduler=(slurm|flux|pbs|shell) -b spec=[(count:(N|max|auto)|duration:T)][,layout:(flat|atomic)][,nodes:(any|same)] PATH
For example, to run the example suite using the shell scheduler in 4 batches:
$ canary run -b scheduler=shell -b spec=count:4 .
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
INFO: Caching test specs
INFO: Created selection 'arcane-axe'
INFO: Selecting test jobs based on runtime environment
INFO: Excluded 14 test jobs
Reason Count
────────────────────────────────────────────────────────────
No single node can accommodate requested resources 14
INFO: Updating live view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
INFO: Starting session 2026-07-18T03-50-30.373008
INFO: Generated 4 test batches from 74 jobs
WARNING: workers=10 > cpu_count=1
INFO: Starting process pool with max 10 workers
Job ID Status Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────────
TestBatch(id=fffecd0) fffecd0 SUBMITTED 1/4
TestBatch(id=fffecd0) fffecd0 STARTED 1/4
TestBatch(id=fffecd0… fffecd0 FAIL (24 SUCCESS, 1 XDIFF, 2 XFAIL, 1 SKIPPE… 18.6s 1/4
TestBatch(id=170e186) 170e186 SUBMITTED 2/4
TestBatch(id=170e186) 170e186 STARTED 2/4
TestBatch(id=170e186… 170e186 FAIL (27 SUCCESS, 2 FAILED, 1 TIMEOUT) 16.1s 2/4
TestBatch(id=a22714f) a22714f SUBMITTED 3/4
TestBatch(id=a22714f) a22714f STARTED 3/4
TestBatch(id=a22714f… a22714f PASS (3 SUCCESS) 3.1s 3/4
TestBatch(id=72ccaa4) 72ccaa4 SUBMITTED 4/4
TestBatch(id=72ccaa4) 72ccaa4 STARTED 4/4
TestBatch(id=72ccaa4… 72ccaa4 PASS (9 SUCCESS) 6.1s 4/4
┌────────────┬─────────┬────────────────┬─────────┬─────────────────────────────────────────────┐
│ Job │ ID │ Status │ Elapsed │ Details │
├────────────┼─────────┼────────────────┼─────────┼─────────────────────────────────────────────┤
│ diff │ 8e033df │ FAIL (DIFFED) │ 0.3s │ Test exited with diff exit code = 64 │
│ skip │ e0e106b │ SKIP (SKIPPED) │ 0.3s │ Test exited with skip exit code = 80 │
│ timeout │ 3afa81a │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
│ xdiff-fail │ 7c452d5 │ FAIL (FAILED) │ 0.3s │ xdiff-fail: expected test to diff │
│ willfail │ cd68ac3 │ FAIL (FAILED) │ 0.1s │ Test exited with exit code = 1 │
│ fail │ de70161 │ FAIL (FAILED) │ 0.3s │ Test exited with exit code = 65 │
│ timeout │ c11972b │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
│ xfail-fail │ 327c2f3 │ FAIL (FAILED) │ 0.3s │ xfail-fail: expected to exit with code != 0 │
└────────────┴─────────┴────────────────┴─────────┴─────────────────────────────────────────────┘
74/74 COMPLETE, 63 SUCCESS, 1 XDIFF, 2 XFAIL, 1 SKIPPED, 1 DIFFED, 4 FAILED, 2 TIMEOUT, in 00:00:43
INFO: Finished session in 44.22 s. with returncode 14
INFO: HTML report written to Canary.html
Note
The shell scheduler submits batches to the user’s shell in a subprocess and is intended for demonstration purposes only. It should not be used outside of running small examples.