Test statuses

Tests can receive one of the following statuses:

pending ready running

success xfail xdiff

failed diffed timeout

blocked skipped cancelled

pending

The test case is waiting for one or more dependencies.

ready

The test case is ready to run.

running

The test case is currently running.

cancelled

The test case was cancelled while running (usually by a keyboard interrupt).

skipped

The test case was skipped due to a skipped dependency or by exiting with a 63 exit code.

Tip

Don’t explicitly exit with code 63. Instead, exit with canary.skip_exit_status or raise a canary.TestSkipped exception.

diffed

A test diffs if it exits with a 64 exit code.

Tip

Don’t explicitly exit with code 64. Instead, exit with canary.diff_exit_status or raise a canary.TestDiffed exception.

failed

A test fails if it exits with any nonzero code not previously defined.

Tip

To explicitly mark a test as failed, exit with canary.fail_exit_status or raise a canary.TestFailed exception.

timeout

The test case exceeded its allowed run time.

success

A test is considered successfully passed if it exits with a 0 exit code.

xfail

The test case is marked as expected to fail

xdiff

The test case is marked as expected to diff

blocked

A test case that was expected to run was not run. Common reasons for being marked blocked are the test case not being run due to a failed or skipped dependency and the test session being stopped prematurely.