Marking a test expected to diff or fail

The canary.directives.xdiff() and canary.directives.xfail() directives can mark tests that you expect to diff or fail, respectively. For example, the following test is expected to diff:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xdiff()


def test():
    raise canary.TestDiffed()


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xdiff.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'taupe-jaguar'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-46-33.131285
INFO: Starting process pool with max 1 workers
Job    ID        Status                                          Elapsed      Rank  
──────────────────────────────────────────────────────────────────────────────────
xdiff  dab99d7   SUBMITTED                                                     1/1  
xdiff  dab99d7   STARTED                                                       1/1  
xdiff  dab99d7   PASS (XDIFF)                                       0.6s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.71 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults

As you can see, the test status was set to xdiff which is considered a successful outcome.

However, if a test that is marked to diff or fail and does not, it will be considered a failure:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xfail()


def test():
    # This test should fail
    return 0


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xfail-fail.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'gentle-dolphin'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-46-34.800242
INFO: Starting process pool with max 1 workers
Job         ID        Status                                          Elapsed      Rank  
───────────────────────────────────────────────────────────────────────────────────────
xfail-fail  327c2f3   SUBMITTED                                                     1/1  
xfail-fail  327c2f3   STARTED                                                       1/1  
xfail-fail  327c2f3   FAIL (FAILED)                                      0.6s       1/1  
┌────────────┬─────────┬───────────────┬─────────┬─────────────────────────────────────────────┐
│ Job        │ ID      │ Status        │ Elapsed │ Details                                     │
├────────────┼─────────┼───────────────┼─────────┼─────────────────────────────────────────────┤
│ xfail-fail │ 327c2f3 │ FAIL (FAILED) │    0.3s │ xfail-fail: expected to exit with code != 0 │
└────────────┴─────────┴───────────────┴─────────┴─────────────────────────────────────────────┘
 1/1 COMPLETE, 1 FAILED, in 00:00:00.65                                                                                 
INFO: Finished session in 0.71 s. with returncode 8
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults

Specifying a nonzero exit code

If a nonzero exit code is expected, use canary.directives.xfail(code), where code is the expected exit code. Any other exit code other than code will be considered a failure.

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.keywords("fast")
canary.directives.xfail(code=23)


def test():
    return 23


if __name__ == "__main__":
    sys.exit(test())

$ canary run ./xstatus/xfail-code.pyt
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from xstatus
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 1 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'radiant-voyage'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-46-36.393485
INFO: Starting process pool with max 1 workers
Job         ID        Status                                          Elapsed      Rank  
───────────────────────────────────────────────────────────────────────────────────────
xfail-code  4ce2ad5   SUBMITTED                                                     1/1  
xfail-code  4ce2ad5   STARTED                                                       1/1  
xfail-code  4ce2ad5   PASS (XFAIL)                                       0.6s       1/1  
INFO: 1/1 tests finished with status PASS
INFO: Finished session in 0.70 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults