Getting help#
canary has several subcommands. To get the list of subcommands, issue
$ canary -h
usage: canary [-hvqd] [--version] [-C path] [-p name] [--banner | --no-banner] [--color {auto,always,never}] [-f file]
[-c path] [-e var=val] [--cache-dir CACHE_DIR] [-r TYPE=N] [--resource-pool-file FILE]
[--enable-hyperthreads] [--gpu-backend GPU_BACKEND]
...
canary - an application testing framework
subcommands:
hpc (batch) Manage and run job batches on an HPC scheduler
view Manage a workspace's view
status Print information about a test run
select Create tagged selection of tests
run Find and run tests from a pathspec
rm Remove Canary workspace
report Create and post test reports
rebaseline Rebaseline tests
log Show the session or a test case's log file
location Print locations of test files and directories
init Initialize a Canary session
info Print information about test session
help Get help on canary and its commands
gc Garbage collect (remove) files generated by cases with status 'success'
find Search paths for test files
fetch Fetch canary assets
edit open test files in $EDITOR
docs open canary documentation in a web browser
describe Print information about a test file, test case
config Get and set configuration options
collect Find and generate test cases
options:
-h, --help show this help message and exit
--version show version and exit
-C path Run as if canary was started in path instead of the current working directory.
-p name Load given plugin module name (multi-allowed). To avoid loading of plugins, use
the `no:` prefix.
-d, --debug Debug mode [default: None]
--color {auto,always,never}
When to color output [default: auto]
--cache-dir CACHE_DIR
Store test case cache info in the given folder [default: .canary_cache/]
console reporting:
-v Increase console logging level by 1
-q Decrease console logging level by 1
--banner, --no-banner
Print banner [default: False]
runtime configuration:
-f file Read configuration settings from this file
-c path Add the colon-separated path to test session's configuration, e.g. -c debug:true
-e var=val Add environment variable var to the testing environment with value val.
resource control:
-r TYPE=N N instances of resource TYPE are available [default: cpus=2]
--resource-pool-file FILE
Read resource pool from this JSON or YAML file
--enable-hyperthreads
Include hyperthreads in resource detection [default: False]
--gpu-backend GPU_BACKEND
Use this GPU backend [default: auto]
more help:
canary help --all list all commands and options
canary help --pathspec help on the path specification syntax
canary docs open https://canary-wm.readthedocs.io/en/production in a browser
To get help on an individual subcommand, issue, for example:
$ canary run -h
usage: canary run [-whRa] [-f file] [-o option] [-k expression] [--owner OWNERS] [-p expression] [--regex regex]
[--tag TAG] [--only {changed,not_pass,failed,not_run,all}] [--fail-fast] [-P {permissive,pedantic}]
[--copy-all-resources] [--empty-ok] [-s CONSOLE_STYLE] [--workers N] [--timeout type=T]
[--no-incremental] [-b option=value] [--hpc-backend BACKEND] [--hpc-scheduler-args ARGS]
[--hpc-batch-spec SPEC] [--hpc-batch-workers WORKERS] [--hpc-batch-timeout-strategy STRATEGY]
[--hpc-queue-timeout T] [--ctest-config cfg] [--ctest-resource-spec-file FILE] [--recurse-ctest]
[--output-on-failure] [--show-excluded-tests] [--oversubscribe TYPE=N] [--no-summary]
[--durations N] [--repeat-until-pass REPEAT_UNTIL_PASS]
[--repeat-after-timeout REPEAT_AFTER_TIMEOUT] [--repeat-until-fail REPEAT_UNTIL_FAIL] [--teardown]
[--mail-to MAIL_TO] [--mail-from MAIL_FROM] [--show-capture [{o,e,oe,no}]]
...
Find and run tests from a pathspec
positional arguments:
pathspec [--] [user args...]
Test file[s] or directories to search. See 'canary help --pathspec' for help on
the path specification
options:
-w Remove test execution directory, if it exists [default: None]
-f file Read test paths from a json or yaml file. See 'canary help --pathfile' for help
on the file schema
--only {changed,not_pass,failed,not_run,all}
Which tests to run after selection
all - run all selected tests, even if already passing
failed - run only previously failing tests
not_run - run tests that have never been executed
changed - run tests that whose specs have newer modification time
not_pass - run tests whose status is not 'SUCCESS' (default)
--fail-fast Stop after first failed test [default: None]
-P {permissive,pedantic}, --parsing-policy {permissive,pedantic}
If pedantic (default), stop if file parsing errors occur, else ignore parsing
errors
--copy-all-resources Do not link resources to the test directory, only copy [default: None]
--empty-ok Exit normally when the test set is empty. By default, an empty test set is an
error (exit code 7)
-s CONSOLE_STYLE, --style CONSOLE_STYLE
Configure live console display style. Given as key=value pairs:
live={yes,no}[yes]: live console updating
name={short,long}[short]: print short (default) names or long
-h, --help Show this help message and exit.
test spec generation:
-o option Turn option(s) on, such as '-o dbg' or '-o intel'
test spec selection:
-k expression Restrict selection to tests matching expression. For example: `-k 'key1 and not
key2'`. The keyword ``:all:`` matches all tests
--owner OWNERS Restrict selection to tests owned by 'owner'
-p expression Restrict selection to tests matching the paramter expression. For example: '-p
cpus=8' or '-p cpus<8'
--regex regex Restrict selection to tests containing the regular expression regex in at least
1 of its file assets. regex is a python regular expression, see
https://docs.python.org/3/library/re.html
--tag TAG Name this selection 'tag'
console reporting:
--show-excluded-tests
Show names of tests that are excluded from the test session False
--no-summary Disable summary [default: False]
--durations N Show N slowest test durations (N<0 for all)
--show-capture [{o,e,oe,no}]
Show captured stdout (o), stderr (e), or both (oe) for failed tests [default:
no]
resource control:
--workers N Execute the test session asynchronously using a pool of at most N workers
--timeout type=T Set the timeout for type (accepts Go's duration format, eg, 40s, 1h20m, 2h,
4h30m30s).
• type=session, the timeout T is applied to the entire test session.
• type=multiplier, the multiplier T is applied to each test's timeout.
• type=*, the timeout T is applied to all test cases.
• type=batch, choices for T are 'conservative' to use a conservative estimate
for batch timeouts (queue times) or 'aggressive'.
Otherwise, a timeout of T is applied to tests having keyword type.
For example, --timeout fast=2 would apply a timeout of 2 seconds to all tests
having the 'fast' keyword; common types are fast, long, default, and ctest.
--no-incremental Don't use the .canary_cache to infer testcase runtimes
--oversubscribe TYPE=N
Apply the multiplier N to the number of slots available per resource instance of
type TYPE
vvtest options:
-R Rerun tests. Normally tests are not run if they previously completed.
-a, --analyze Only run the analysis sections of each test. Note that a test must be written to
support this option (using the vvtest_util.is_analysis_only flag) otherwise the
whole test is run.
canary hpc:
-b option=value Short cut for setting batch options.
--hpc-backend BACKEND, --scheduler BACKEND
Submit batches to this HPC scheduler [alias: -b backend=BACKEND] [default: None]
--hpc-scheduler-args ARGS
Comma separated list of options to pass directly to the scheduler [alias: -b
options=ARGS]
--hpc-batch-spec SPEC
Comma separated list of options to partition test cases into batches. See canary
batch help --spec for help on batch specification syntax [alias: -b spec=SPEC]
--hpc-batch-workers WORKERS
Run test cases in batches using WORKERS workers [alias: -b workers=WORKERS]
--hpc-batch-timeout-strategy STRATEGY
Estimate batch runtime (queue time) conservatively or aggressively [alias: -b
timeout=STRATEGY] [default: aggressive]
--hpc-queue-timeout T
Maximum time to wait in queue [alias: -b queue_timeout=T] [default: 30min]
ctest options:
--ctest-config cfg Choose configuration to test
--ctest-resource-spec-file FILE
Set the resource spec file to use.
--recurse-ctest Recurse CMake binary directory for test files. CTest tests can be detected from
the root CTestTestfile.cmake, so this is option is not necessary unless there is
a mix of CTests and other test types in the binary directory
--output-on-failure Alias for --show-capture
repeat:
--repeat-until-pass REPEAT_UNTIL_PASS
Allow each test to run up to <n> times in order to pass
--repeat-after-timeout REPEAT_AFTER_TIMEOUT
Allow each test to run up to <n> times if it times out
--repeat-until-fail REPEAT_UNTIL_FAIL
Require each test to run <n> times without failing in order to pass
plugin options:
--teardown, --post-clean
Clean up files created by a test if it finishes successfully [default: None]
--mail-to MAIL_TO Send a test session summary to the comma separated list of email addresses
--mail-from MAIL_FROM
Send mail from this user
See canary help --pathspec for help on the path specification