Running tests in a batch scheduler#
Execution in a batch scheduler is accomplished by defining the scheduler and batching scheme on the command line:
canary run -b scheduler=(slurm|flux|pbs|shell) -b spec=[(count:(N|max|auto)|duration:T)][,layout:(flat|atomic)][,nodes:(any|same)] PATH
For example, to run the example suite using the shell scheduler in 4 batches:
$ canary run -b scheduler=shell -b spec=count:4 .
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
INFO: Caching test specs
INFO: Created selection 'arctic-viper'
INFO: Selecting test jobs based on runtime environment
INFO: Excluded 13 test jobs
Reason Count
────────────────────────────────────
insufficient slots of cpus 10
Resource unavailable: gpus 3
INFO: Starting session 2026-06-04T20-42-27.970921
INFO: Generated 4 test batches from 67 jobs
WARNING: workers=10 > cpu_count=1
INFO: Starting process pool with max 10 workers
Job ID Status Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────────
TestBatch(id=bf0ba71) bf0ba71 SUBMITTED 1/4
TestBatch(id=bf0ba71) bf0ba71 STARTED 1/4
TestBatch(id=bf0ba71… bf0ba71 FAIL (21 SUCCESS, 2 XFAIL, 1 SKIPPED, 3 FAIL… 17.7s 1/4
TestBatch(id=69bd323) 69bd323 SUBMITTED 2/4
TestBatch(id=69bd323) 69bd323 STARTED 2/4
TestBatch(id=69bd323… 69bd323 FAIL (23 SUCCESS, 1 XDIFF, 1 DIFFED, 1 FAILE… 14.1s 2/4
TestBatch(id=4845aa8) 4845aa8 SUBMITTED 3/4
TestBatch(id=4845aa8) 4845aa8 STARTED 3/4
TestBatch(id=4845aa8… 4845aa8 PASS (3 SUCCESS) 3.0s 3/4
TestBatch(id=92e7fdc) 92e7fdc SUBMITTED 4/4
TestBatch(id=92e7fdc) 92e7fdc STARTED 4/4
TestBatch(id=92e7fdc… 92e7fdc PASS (9 SUCCESS) 6.1s 4/4
┌────────────┬─────────┬────────────────┬─────────┬─────────────────────────────────────────────┐
│ Job │ ID │ Status │ Elapsed │ Details │
├────────────┼─────────┼────────────────┼─────────┼─────────────────────────────────────────────┤
│ fail │ de70161 │ FAIL (FAILED) │ 0.3s │ Test exited with exit code = 65 │
│ skip │ e0e106b │ SKIP (SKIPPED) │ 0.3s │ Test exited with skip exit code = 80 │
│ timeout │ 3afa81a │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
│ xfail-fail │ 327c2f3 │ FAIL (FAILED) │ 0.3s │ xfail-fail: expected to exit with code != 0 │
│ willfail │ cd68ac3 │ FAIL (FAILED) │ 0.1s │ Test exited with exit code = 1 │
│ diff │ 8e033df │ FAIL (DIFFED) │ 0.3s │ Test exited with diff exit code = 64 │
│ timeout │ c11972b │ FAIL (TIMEOUT) │ 2.2s │ Job timed out after 2.0 s. │
│ xdiff-fail │ 7c452d5 │ FAIL (FAILED) │ 0.3s │ xdiff-fail: expected test to diff │
└────────────┴─────────┴────────────────┴─────────┴─────────────────────────────────────────────┘
67/67 COMPLETE, 56 SUCCESS, 1 XDIFF, 2 XFAIL, 1 SKIPPED, 1 DIFFED, 4 FAILED, 2 TIMEOUT, in 00:00:40
INFO: Finished session in 41.61 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
Note
The shell scheduler submits batches to the user’s shell in a subprocess and is intended for demonstration purposes only. It should not be used outside of running small examples.