Run CTest tests

In addition to CMake integration, canary can run CTest tests natively. Simply pass the path to a CMake build directory containing CTest tests:

$ canary run CMAKE_BINARY_DIR

canary will read the CTest instructions for each test added by add_test and run the test.

Supported CTest properties

The following CTest properties are supported by canary.

• ATTACHED_FILES

• ATTACHED_FILES_ON_FAIL

• DEPENDS

• DISABLED

• ENVIRONMENT

• ENVIRONMENT_MODIFICATION

• FAIL_REGULAR_EXPRESSION

• FIXTURES_CLEANUP

• FIXTURES_REQUIRED

• FIXTURES_SETUP

• LABELS

• PASS_REGULAR_EXPRESSION

• PROCESSORS

• RESOURCE_GROUPS

• RUN_SERIAL

• SKIP_REGULAR_EXPRESSION

• SKIP_RETURN_CODE

• TIMEOUT

• WILL_FAIL

• WORKING_DIRECTORY

Unsupported CTest properties

The following CTest properties are not supported by canary.

• COST

• GENERATED_RESOURCE_SPEC_FILE

• MEASUREMENT

• PROCESSOR_AFFINITY

• REQUIRED_FILES

• RESOURCE_LOCK

• TIMEOUT_AFTER_MATCH

• TIMEOUT_SIGNAL_GRACE_PERIOD

• TIMEOUT_SIGNAL_NAME

Note

Contact the canary developers if you require any one of these properties to run your test suite.

Differences in behavior from CTest

DEPENDS

The CTest DEPENDS property sets the execution order, but “the results of those tests are not considered, the dependency relationship is purely for order of execution”. In canary, the results of the tests are considered. See depends_on.

RESOURCE_GROUPS

canary supports the CTest resource specification file. See Resource allocation.

*_REGULAR_EXPRESSION behavior

*_REGULAR_EXPRESSION patterns are evaluated in the following order:

If PASS_REGULAR_EXPRESSION is defined

set status to success if any pass regular expression matches, otherwise set status to failed.

If SKIP_RETURN_CODE is defined

set status to skipped if the test’s return code is equal to SKIP_RETURN_CODE

If SKIP_REGULAR_EXPRESSION is defined

set status to skipped if any skip regular expression matches

If FAIL_REGULAR_EXPRESSION is defined

set status to failed if any fail regular expression matches

Thus, if a test defines both PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION and the output contains both patterns, the test will be marked as failed since the fail regular expression is evaluated last.