The execute and analyze pattern#
The “execute and analyze” pattern generates a collection of test cases consisting of
a test file’s parameterized instantiations; and
the test file’s base, un-parameterized, case.
The base case runs only after all of the parameterized test cases are finished.
The execute and analyze pattern is enabled by adding canary.directives.generate_composite_base_case() to the test file’s directives.
vvtest compatibility
In vvtest, the name of this directive is analyze
Consider the directives section of the test file examples/execute_and_analyze/execute_and_analyze.pyt:
import sys
import canary
canary.directives.parameterize("a", [1, 2, 3])
canary.directives.generate_composite_base_case()
The dependency graph for this test is
$ canary describe execute_and_analyze/execute_and_analyze.pyt
--- execute_and_analyze ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/execute_and_analyze/execute_and_analyze.pyt
Keywords:
4 test specs using on_options=:
└── execute_and_analyze
│ ├── execute_and_analyze.a=1
│ ├── execute_and_analyze.a=2
│ └── execute_and_analyze.a=3
As can be seen, the base case execute_and_analyze depends on execute_and_analyze.a=1, execute_and_analyze.a=2, and execute_and_analyze.a=3. When the test is run, these “children” tests are run first and then the base case:
$ canary run ./execute_and_analyze
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from execute_and_analyze
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 4 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'sand-castle'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-34-57.890275
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
execute_and_analyze.a=3 7f17379 SUBMITTED 2/4
execute_and_analyze.a=3 7f17379 STARTED 0.3s 2/4
execute_and_analyze.a=1 eb3d5cf SUBMITTED 1/4
execute_and_analyze.a=1 eb3d5cf STARTED 0.3s 1/4
execute_and_analyze.a=3 7f17379 PASS (SUCCESS) 0.2s 0.5s 2/4
execute_and_analyze.a=1 eb3d5cf PASS (SUCCESS) 0.3s 0.5s 1/4
execute_and_analyze.a=2 f2c9df6 SUBMITTED 3/4
execute_and_analyze.a=2 f2c9df6 STARTED 0.0s 3/4
execute_and_analyze.a=2 f2c9df6 PASS (SUCCESS) 0.0s 0.2s 3/4
execute_and_analyze 4c59603 SUBMITTED 4/4
execute_and_analyze 4c59603 STARTED 0.0s 4/4
execute_and_analyze 4c59603 PASS (SUCCESS) 0.0s 0.2s 4/4
INFO: 4/4 tests finished with status PASS
INFO: Finished session in 1.02 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults
Test execution phases#
To take advantage of the execute and analyze pattern, a test should define separate functions for the parameterized and base test cases. For example, a test might define separate run_parameterized_case and analyze_composite_base_case functions as below:
def run_parameterized_case(case: canary.TestInstance) -> None:
# Run the test
f = f"{case.parameters.a}.txt"
canary.filesystem.touchp(f)
def analyze_composite_base_case(case: canary.TestMultiInstance) -> None:
# Analyze the collective
assert len(case.dependencies) == 3
for dep in case.dependencies:
f = os.path.join(dep.working_directory, f"{dep.parameters.a}.txt")
assert os.path.exists(f)
The function run_parameterized_test is intended to be called for each parameterized child test and the function analyze_composite_base_case the final composite base case (in which the children tests are made available in the canary.TestMultiInstance.dependencies attribute).
You can key off of the test instance type to determine which function to call:
def main():
self = canary.get_instance()
if isinstance(self, canary.TestMultiInstance):
analyze_composite_base_case(self)
else:
run_parameterized_case(self)
return 0
The full example#
# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT
import os
import sys
import canary
canary.directives.parameterize("a", [1, 2, 3])
canary.directives.generate_composite_base_case()
def run_parameterized_case(case: canary.TestInstance) -> None:
# Run the test
f = f"{case.parameters.a}.txt"
canary.filesystem.touchp(f)
def analyze_composite_base_case(case: canary.TestMultiInstance) -> None:
# Analyze the collective
assert len(case.dependencies) == 3
for dep in case.dependencies:
f = os.path.join(dep.working_directory, f"{dep.parameters.a}.txt")
assert os.path.exists(f)
def main():
self = canary.get_instance()
if isinstance(self, canary.TestMultiInstance):
analyze_composite_base_case(self)
else:
run_parameterized_case(self)
return 0
if __name__ == "__main__":
sys.exit(main())
Accessing dependency parameters#
Dependency parameters can be accessed directly from the base test instance’s dependencies, e.g.,
self = canary.get_instance()
self.dependencies[0].parameters
or, in the base test instance’s parameters attribute. Consider the following test:
import os
import sys
import canary
The parameters cpus, a, and b of each dependency can be accessed directly:
def composite_base_case(case: canary.TestMultiInstance):
for dep in case.dependencies:
The ordering of the parameters is guaranteed to be the same as the ordering the dependencies. E.g., self.dependencies[i].parameters.a == self.parameters.a[i].
Additionally, a full table of dependency parameters is accessible via key entry into the parameters attribute, where the key is a tuple containing each individual parameter name, e.g.:
assert parameters["cpus"] == dep.parameters.cpus
assert parameters["a"] == dep.parameters.a
assert parameters["b"] == dep.parameters.b
assert sorted(case.parameters.cpus) == sorted((1, 1, 1, 2, 2, 2))
assert sorted(case.parameters.a) == sorted((1, 2, 4, 1, 2, 4))
assert sorted(case.parameters.b) == sorted((2, 3, 5, 2, 3, 5))
assert sorted(case.parameters[("cpus", "a", "b")]) == sorted(
(
Run only the analysis section of a test#
After the test has been run, the analysis sections can be run without rerunning the (potentially expensive) test portion, navigate the test’s results directory and execute canary run:
$ cd $(canary location execute_and_analyze); canary run .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 4 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'sharp-dawn'
INFO: Selecting test cases based on runtime environment
INFO: Starting session 2026-04-21T15-35-00.132792
INFO: Starting process pool with max 2 workers
Job ID Status Queued Elapsed Rank
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
execute_and_analyze.a=3 840c8d4 SUBMITTED 2/4
execute_and_analyze.a=3 840c8d4 STARTED 0.3s 2/4
execute_and_analyze.a=1 e91d48d SUBMITTED 1/4
execute_and_analyze.a=1 e91d48d STARTED 0.3s 1/4
execute_and_analyze.a=1 e91d48d PASS (SUCCESS) 0.3s 0.5s 1/4
execute_and_analyze.a=2 0cde3e1 SUBMITTED 3/4
execute_and_analyze.a=2 0cde3e1 STARTED 0.0s 3/4
execute_and_analyze.a=3 840c8d4 PASS (SUCCESS) 0.3s 0.6s 2/4
execute_and_analyze.a=2 0cde3e1 PASS (SUCCESS) 0.0s 0.2s 3/4
execute_and_analyze 2547073 SUBMITTED 4/4
execute_and_analyze 2547073 STARTED 0.0s 4/4
execute_and_analyze 2547073 PASS (SUCCESS) 0.0s 0.2s 4/4
INFO: 4/4 tests finished with status PASS
INFO: Finished session in 1.01 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples/TestResults