The execute and analyze pattern#
The “execute and analyze” pattern generates a collection of jobs consisting of
a test file’s parameterized instantiations; and
the test file’s base, un-parameterized, job.
The base case runs only after all of the parameterized jobs are finished.
The execute and analyze pattern is enabled by adding canary.directives.generate_composite_base_case() to the test file’s directives.
vvtest compatibility
In vvtest, the name of this directive is analyze
Consider the directives section of the test file examples/execute_and_analyze/execute_and_analyze.pyt:
import sys
import canary
canary.directives.parameterize("a", [1, 2, 3])
canary.directives.generate_composite_base_case()
The dependency graph for this test is
$ canary describe execute_and_analyze/execute_and_analyze.pyt
--- execute_and_analyze ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/execute_and_analyze/execute_and_analyze.pyt
Keywords:
4 test specs using on_options=:
├── execute_and_analyze.a=1
├── execute_and_analyze.a=2
├── execute_and_analyze.a=3
└── execute_and_analyze
│ ├── execute_and_analyze.a=1
│ ├── execute_and_analyze.a=2
│ └── execute_and_analyze.a=3
As can be seen, the base case execute_and_analyze depends on execute_and_analyze.a=1, execute_and_analyze.a=2, and execute_and_analyze.a=3. When the test is run, these “children” tests are run first and then the base case:
$ canary run ./execute_and_analyze
INFO: Initializing empty canary workspace at .
INFO: Collecting generator files from execute_and_analyze
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 4 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'gunmetal-willow'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-43-55.585316
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────────────
execute_and_analyze.a=1 42d307a SUBMITTED 1/4
execute_and_analyze.a=1 42d307a STARTED 1/4
execute_and_analyze.a=1 42d307a PASS (SUCCESS) 0.7s 1/4
execute_and_analyze.a=3 7be73db SUBMITTED 2/4
execute_and_analyze.a=3 7be73db STARTED 2/4
execute_and_analyze.a=3 7be73db PASS (SUCCESS) 0.4s 2/4
execute_and_analyze.a=2 1cc474b SUBMITTED 3/4
execute_and_analyze.a=2 1cc474b STARTED 3/4
execute_and_analyze.a=2 1cc474b PASS (SUCCESS) 0.4s 3/4
execute_and_analyze 9ffee8c SUBMITTED 4/4
execute_and_analyze 9ffee8c STARTED 4/4
execute_and_analyze 9ffee8c PASS (SUCCESS) 0.4s 4/4
INFO: 4/4 tests finished with status PASS
INFO: Finished session in 2.17 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults
Test execution phases#
To take advantage of the execute and analyze pattern, a test should define separate functions for the parameterized and base cases. For example, a test might define separate run_parameterized_job and analyze_composite_base_job functions as below:
def run_parameterized_case(case: canary.TestInstance) -> None:
# Run the test
f = f"{case.parameters.a}.txt"
canary.filesystem.touchp(f)
def analyze_composite_base_case(case: canary.TestMultiInstance) -> None:
# Analyze the collective
assert len(case.dependencies) == 3
for dep in case.dependencies:
f = os.path.join(dep.working_directory, f"{dep.parameters.a}.txt")
assert os.path.exists(f)
The function run_parameterized_test is intended to be called for each parameterized child test and the function analyze_composite_base_job the final composite base case (in which the children tests are made available in the canary.TestMultiInstance.dependencies attribute).
You can key off of the test instance type to determine which function to call:
def main():
self = canary.get_instance()
if isinstance(self, canary.TestMultiInstance):
analyze_composite_base_case(self)
else:
run_parameterized_case(self)
return 0
The full example#
# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT
import os
import sys
import canary
canary.directives.parameterize("a", [1, 2, 3])
canary.directives.generate_composite_base_case()
def run_parameterized_case(case: canary.TestInstance) -> None:
# Run the test
f = f"{case.parameters.a}.txt"
canary.filesystem.touchp(f)
def analyze_composite_base_case(case: canary.TestMultiInstance) -> None:
# Analyze the collective
assert len(case.dependencies) == 3
for dep in case.dependencies:
f = os.path.join(dep.working_directory, f"{dep.parameters.a}.txt")
assert os.path.exists(f)
def main():
self = canary.get_instance()
if isinstance(self, canary.TestMultiInstance):
analyze_composite_base_case(self)
else:
run_parameterized_case(self)
return 0
if __name__ == "__main__":
sys.exit(main())
Accessing dependency parameters#
Dependency parameters can be accessed directly from the base test instance’s dependencies, e.g.,
self = canary.get_instance()
self.dependencies[0].parameters
or, in the base test instance’s parameters attribute. Consider the following test:
import os
import sys
import canary
The parameters cpus, a, and b of each dependency can be accessed directly:
def composite_base_case(case: canary.TestMultiInstance):
for dep in case.dependencies:
The ordering of the parameters is guaranteed to be the same as the ordering the dependencies. E.g., self.dependencies[i].parameters.a == self.parameters.a[i].
Additionally, a full table of dependency parameters is accessible via key entry into the parameters attribute, where the key is a tuple containing each individual parameter name, e.g.:
assert parameters["cpus"] == dep.parameters.cpus
assert parameters["a"] == dep.parameters.a
assert parameters["b"] == dep.parameters.b
assert sorted(case.parameters.cpus) == sorted((1, 1, 1, 2, 2, 2))
assert sorted(case.parameters.a) == sorted((1, 2, 4, 1, 2, 4))
assert sorted(case.parameters.b) == sorted((2, 3, 5, 2, 3, 5))
assert sorted(case.parameters[("cpus", "a", "b")]) == sorted(
(
Run only the analysis section of a test#
After the test has been run, the analysis sections can be run without rerunning the (potentially expensive) test portion, navigate the test’s results directory and execute canary run:
$ cd $(canary location execute_and_analyze); canary run .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 4 test specs from 1 generators
INFO: Caching test specs
INFO: Created selection 'hidden-fjord'
INFO: Selecting test jobs based on runtime environment
INFO: Starting session 2026-06-04T20-43-59.248488
INFO: Starting process pool with max 1 workers
Job ID Status Elapsed Rank
────────────────────────────────────────────────────────────────────────────────────────────────────
execute_and_analyze.a=1 86e878c SUBMITTED 1/4
execute_and_analyze.a=1 86e878c STARTED 1/4
execute_and_analyze.a=1 86e878c PASS (SUCCESS) 0.7s 1/4
execute_and_analyze.a=3 73712b4 SUBMITTED 2/4
execute_and_analyze.a=3 73712b4 STARTED 2/4
execute_and_analyze.a=3 73712b4 PASS (SUCCESS) 0.4s 2/4
execute_and_analyze.a=2 a4a191f SUBMITTED 3/4
execute_and_analyze.a=2 a4a191f STARTED 3/4
execute_and_analyze.a=2 a4a191f PASS (SUCCESS) 0.4s 3/4
execute_and_analyze 698deab SUBMITTED 4/4
execute_and_analyze 698deab STARTED 4/4
execute_and_analyze 698deab PASS (SUCCESS) 0.4s 4/4
INFO: 4/4 tests finished with status PASS
INFO: Finished session in 2.17 s. with returncode 0
INFO: Updating view at /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/TestResults