Marking a test expected to diff or fail

The canary.directives.xdiff() and canary.directives.xfail() directives can mark tests that you expect to diff or fail, respectively. For example, the following test is expected to diff:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xdiff()


def test():
    raise canary.TestDiffed()


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xdiff.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'candid-lobo'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-40.280028
INFO: Starting process pool with max 2 workers
Job    ID        Status                                           Queued   Elapsed      Rank  
────────────────────────────────────────────────────────────────────────────────────────────
xdiff  3389dae   SUBMITTED                                                               1/1  
xdiff  3389dae   STARTED                                            0.3s                 1/1  
xdiff  3389dae   PASS (XDIFF)                                       0.3s      0.5s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.53 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults

As you can see, the test status was set to xdiff which is considered a successful outcome.

However, if a test that is marked to diff or fail and does not, it will be considered a failure:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xfail()


def test():
    # This test should fail
    return 0


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xfail-fail.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'brave-garden'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-41.537895
INFO: Starting process pool with max 2 workers
Job         ID        Status                                           Queued   Elapsed      Rank  
─────────────────────────────────────────────────────────────────────────────────────────────────
xfail-fail  9db7d1b   SUBMITTED                                                               1/1  
xfail-fail  9db7d1b   STARTED                                            0.3s                 1/1  
xfail-fail  9db7d1b   FAIL (FAILED)                                      0.3s      0.5s       1/1  
┌──────────────┬────────────┬────────────────┬───────────┬───────────┬─────────────────────────────────────────────────┐
│ Job          │ ID         │ Status         │    Queued │   Elapsed │ Details                                         │
├──────────────┼────────────┼────────────────┼───────────┼───────────┼─────────────────────────────────────────────────┤
│ xfail-fail   │ 9db7d1b    │ FAIL (FAILED)  │      0.0s │      0.2s │ xfail-fail: expected to exit with code != 0     │
└──────────────┴────────────┴────────────────┴───────────┴───────────┴─────────────────────────────────────────────────┘
 1/1 COMPLETE, 1 FAILED, in 00:00:00.47                                                                                 
INFO: Finished session in 0.53 s. with returncode 8
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults

Specifying a nonzero exit code

If a nonzero exit code is expected, use canary.directives.xfail(code), where code is the expected exit code. Any other exit code other than code will be considered a failure.

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xfail(code=23)


def test():
    return 23


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xfail-code.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'lime-veil'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-42.800338
INFO: Starting process pool with max 2 workers
Job         ID        Status                                           Queued   Elapsed      Rank  
─────────────────────────────────────────────────────────────────────────────────────────────────
xfail-code  80a56f0   SUBMITTED                                                               1/1  
xfail-code  80a56f0   STARTED                                            0.3s                 1/1  
xfail-code  80a56f0   PASS (XFAIL)                                       0.3s      0.5s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.53 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults