Running tests in a batch scheduler#
Execution in a batch scheduler is accomplished by defining the scheduler and batching scheme on the command line:
canary run -b scheduler=(slurm|flux|pbs|shell) -b spec=[(count:(N|max|auto)|duration:T)][,layout:(flat|atomic)][,nodes:(any|same)] PATH
For example, to run the example suite using the shell scheduler in 4 batches:
$ canary run -b scheduler=shell -b spec=count:4 .
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
INFO: Caching test specs
INFO: Created selection 'dapper-puma'
INFO: Selecting test cases based on runtime environment
INFO: Excluded 13 test cases
Reason Count
────────────────────────────────────
insufficient slots of cpus 10
Resource unavailable: gpus 3
INFO: Starting session 2026-04-21T15-34-08.325202
INFO: Generated 4 test batches from 67 test cases
WARNING: workers=10 > cpu_count=2
INFO: Starting process pool with max 10 workers
Job ID Status Queued Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────────────────────
TestBatch(id=25e1801) 25e1801 SUBMITTED 2/4
TestBatch(id=ef124ae) ef124ae SUBMITTED 1/4
TestBatch(id=25e1801) 25e1801 STARTED 2.4s 2/4
TestBatch(id=ef124ae) ef124ae STARTED 2.4s 1/4
TestBatch(id=25e1801) 25e1801 NONE (16 PASS, 2 FAILED, 1 TIMEOUT) 2.4s 9.4s 2/4
TestBatch(id=ef124ae) ef124ae NONE (18 PASS, 1 TIMEOUT) 2.4s 9.4s 1/4
TestBatch(id=454fd0f) 454fd0f SUBMITTED 3/4
TestBatch(id=454fd0f) 454fd0f STARTED 1.0s 3/4
TestBatch(id=454fd0f) 454fd0f NONE (16 PASS, 1 SKIPPED, 1 DIFFED, 2 FAILED) 1.0s 4.0s 3/4
TestBatch(id=27057cf) 27057cf SUBMITTED 4/4
TestBatch(id=27057cf) 27057cf STARTED 1.0s 4/4
TestBatch(id=27057cf) 27057cf NONE (9 PASS) 1.0s 2.0s 4/4
┌──────────────┬────────────┬────────────────┬───────────┬───────────┬─────────────────────────────────────────────────┐
│ Job │ ID │ Status │ Queued │ Elapsed │ Details │
├──────────────┼────────────┼────────────────┼───────────┼───────────┼─────────────────────────────────────────────────┤
│ timeout │ 1c2f507 │ FAIL (TIMEOUT) │ 0.0s │ 2.2s │ Job timed out after 2.0 s. │
│ xdiff-fail │ f4263d4 │ FAIL (FAILED) │ 0.0s │ 0.3s │ xdiff-fail: expected test to diff │
│ xfail-fail │ 9db7d1b │ FAIL (FAILED) │ 0.0s │ 0.4s │ xfail-fail: expected to exit with code != 0 │
│ timeout │ 7127eb6 │ FAIL (TIMEOUT) │ 0.0s │ 2.2s │ Job timed out after 2.0 s. │
│ diff │ b4597e3 │ FAIL (DIFFED) │ 0.0s │ 0.2s │ Test exited with diff exit code = 64 │
│ fail │ a850e81 │ FAIL (FAILED) │ 0.0s │ 0.2s │ Test exited with exit code = 65 │
│ skip │ e01f382 │ SKIP (SKIPPED) │ 0.0s │ 0.2s │ Test exited with skip exit code = 80 │
│ willfail │ e4caa24 │ FAIL (FAILED) │ 0.0s │ 0.1s │ Test exited with exit code = 1 │
└──────────────┴────────────┴────────────────┴───────────┴───────────┴─────────────────────────────────────────────────┘
67/67 COMPLETE, 56 SUCCESS, 1 XDIFF, 2 XFAIL, 1 SKIPPED, 1 DIFFED, 4 FAILED, 2 TIMEOUT, in 00:00:15
INFO: Finished session in 15.93 s. with returncode 14
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Note
The shell scheduler submits batches to the user’s shell in a subprocess and is intended for demonstration purposes only. It should not be used outside of running small examples.