Getting started

canary identifies and executes tests defined from multiple sources. Each test runs in its own execution directory and return code captured. A return code of 0 indicates success, while any other return code indicates a failure of some kind. Python files having a .pyt extension are the native test format which will be used in the examples.

Note

If canary is not installed on your system, you can install it as:

python3 -m venv venv
source ./venv/bin/activate
python3 -m pip install canary-wm

Note

All of the examples in the documentation can be obtained by the canary fetch command:

canary fetch examples

A first test

The test file first.pyt defines a function that adds two numbers and verifies it for correctness:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("basic")


def add(a: int, b: int) -> int:
    return a + b


def test():
    assert add(3, 2) == 5


if __name__ == "__main__":
    sys.exit(test())

To run the test, navigate to the examples directory and run canary run -k first ./basic which tells canary to run tests found in the path ./basic and to filter tests to include only those tests with the keyword first:

$ canary -d run -k first ./basic/first
DEBUG: Loading Canary workspace from /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from basic/first
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Selecting specs based on 1 rules
INFO: Created selection 'hidden-orchard'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-43-35.087762
DEBUG: Job 4b172fc added to queue with cost 300.00166666203705
INFO: Starting process pool with max 1 workers
Job    ID        Status                                          Elapsed      Rank  
──────────────────────────────────────────────────────────────────────────────────
first  4b172fc   SUBMITTED                                                     1/1  
first  4b172fc   STARTED                                                       1/1  
first  4b172fc   PASS (SUCCESS)                                     0.6s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.71 s. with returncode 0

✨✨ Session done -- 1 total: 1 pass in 00:00:00.70
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults

A test is considered to have successfully completed if its exit code is 0. See Test statuses for more details on test statuses.

Note

This test uses an optional keyword directive to aid in identifying the test and is used to filter tests (-k first on the command line).

canary creates an isolated workspace at the start of each run inside the .canary folder. All inputs, intermediate files, and outputs are contained within the workspace. Once test execution completes, a “view” of the most recent results is created in the TestResults directory. Details of the latest results are seen by canary status:

$ canary status
┌────────┬────────────┬──────────────────┬──────────────────────┬────────────────────┬────────────────┬────────────────┐
│ ID     │ Name       │ Session          │ Exit Code            │ Duration           │ Status         │ Details        │
├────────┼────────────┼──────────────────┼──────────────────────┼────────────────────┼────────────────┼────────────────┤
└────────┴────────────┴──────────────────┴──────────────────────┴────────────────────┴────────────────┴────────────────┘

By default, only failed tests appear in the output. To see the results of each test in the session, including passed tests, pass -rA:

$ canary status -rA
┌────────────┬─────────┬───────────────────────────────────┬──────────────┬────────────┬────────────────────┬──────────┐
│ ID         │ Name    │ Session                           │ Exit Code    │ Duration   │ Status             │ Details  │
├────────────┼─────────┼───────────────────────────────────┼──────────────┼────────────┼────────────────────┼──────────┤
│ 4b172fc    │ first   │ 2026-06-04T20-43-35.087762        │ 0            │ 0.31       │ PASS (SUCCESS)     │          │
└────────────┴─────────┴───────────────────────────────────┴──────────────┴────────────┴────────────────────┴──────────┘

A second test

In this second example, the external program “add.py” adds two numbers and writes the result to the console’s stdout is tested.

#!/usr/bin/env python
# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT
import argparse


def add(a: int, b: int) -> int:
    return a + b


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("a", type=int)
    p.add_argument("b", type=int)
    args = p.parse_args()
    print(add(args.a, args.b))

In the test, add.py is linked to the execution directory, is executed, and output verified for correctness:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("basic", "second")
canary.directives.link("add.py")


def test():
    print("Verifying that 2 + 3 = 5")
    import os

    print(os.getcwd())
    add = canary.Executable(f"{sys.executable} ./add.py")
    result = add("2", "3", stdout=str)
    assert int(result.get_output()) == 5, "Bummer, test failed."
    print("Test passed!")


if __name__ == "__main__":
    sys.exit(test())

This test introduces two new features:

• canary.directives.link(): links add.py into the execution directory (see Test Directives for more directives); and

• canary.Executable: creates a callable wrapper around executable scripts.

To run the test, navigate to the examples folder and run:

$ canary run -k second ./basic/second
INFO: Collecting generator files from basic/second
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Selecting specs based on 1 rules
INFO: Created selection 'navy-current'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-43-37.876552
INFO: Starting process pool with max 1 workers
Job     ID        Status                                          Elapsed      Rank  
───────────────────────────────────────────────────────────────────────────────────
second  b1f18b4   SUBMITTED                                                     1/1  
second  b1f18b4   STARTED                                                       1/1  
second  b1f18b4   PASS (SUCCESS)                                     0.6s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.70 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults

Inspecting test output

When a test is run, its output is captured to the file canary-out.txt in its execution directory. The canary log command can find and print the contents of this file to the console:

$ canary log second
/home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second/canary-out.txt:
[2026-06-04-20:43:38.196040] Creating workspace root at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second
[2026-06-04-20:43:38.196207] Preparing test: second
[2026-06-04-20:43:38.196207] Directory: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second
[2026-06-04-20:43:38.196207] Linking and copying working files...
[2026-06-04-20:43:38.196207] Linking /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/basic/second/second.pyt to /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second
[2026-06-04-20:43:38.196207] Linking /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/basic/second/add.py to /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second
[2026-06-04-20:43:38.198835] Begin executing second
Verifying that 2 + 3 = 5
/home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/.canary/sessions/2026-06-04T20-43-37.876552/second
Test passed!