Parameterizing tests

A single test file can generate many test cases, each having different parameters, using the parameterize directive. The test file uses the parameter name[s] and value[s] to run variations of the test. For example

canary.directives.parameterize("odd,even", [(1, 2), (3, 4)])

instructs canary to create two test instances with parameters odd=1 and even=2 in the first, and parameters odd=3 and even=4 in the second.

Special parameter names

The cpus and gpus parameters are interpreted by canary to be the number of cpus and gpus, respectively, needed by the test case.

vvtest compatiblity

The np and ndevice parameters are taken to be synonyms for cpus and gpus, respectively.

The type argument

parameterize takes an optional argument type, allowing parameters to be generated in different ways from the input values. Three types are recoginized:

• list_parameter_space (default)

• centered_parameter_space

• random_parameter_space

type=canary.list_parameter_space

def parameterize(
    names: str | tuple[str],
    values: Iterable[Iterable[str | float | int]],
    when: dict | str,
    type: canary.enums = canary.list_parameter_space
)

The list_parameter_space reads lists of parameter values as defined by the user. names is a comma-separated list of names, or tuple of names. values is the list of the associated values such that len(values[i]) are the parameter values for the i’th generated test case. Consequently, len(names) == len(values[i]) for all i. For example,

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.parameterize("a", (1, 4))


def test():
    self = canary.get_instance()
    print(f"{self.parameters.a}")
    return 0


if __name__ == "__main__":
    sys.exit(test())

will produce two test cases, one with a=1 and another with a=4, each executed in their own test directory:

$ canary describe parameterize/parameterize1.pyt
--- parameterize1 ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/parameterize/parameterize1.pyt
Keywords: 
2 test specs using on_options=:
├── parameterize1.a=1
└── parameterize1.a=4

Multiple parameter names and their values can be defined:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.parameterize("a,b", [(1, 2), (5, 6)])


def test():
    self = canary.test.instance
    print(f"{self.parameters.a=}, {self.parameters.b=}")


if __name__ == "__main__":
    sys.exit(test())

which would result in the following two tests

$ canary describe parameterize/parameterize2.pyt
--- parameterize2 ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/parameterize/parameterize2.pyt
Keywords: 
2 test specs using on_options=:
├── parameterize2.a=1.b=2
└── parameterize2.a=5.b=6

type=canary.centered_parameter_space

def parameterize(
    names: str | tuple[str],
    values: Iterable[Iterable[str | float | int]],
    samples: int = 10,
    when: dict | str,
    type: canary.enums = canary.centered_parameter_space
)

The centered_parameter_space type computes parameter sets along multiple coordinate-based vectors, one per parameter, centered about the initial values. names is a tuple or comma-separated string of parameter names and values is a list of tuples where values[i] = (initial_value, step_size, num_steps) define the initial value, step size, and number of steps for the ith parameter.

The capability is modeled after the capability of the same name in Dakota.

The centered parameter space takes steps along each orthogonal dimension. Each dimension is treated independently. The number of steps are taken in each direction, so that the total number of points in the parameter study is \(1+ 2\sum{n}\).

Example

import json
import os
import sys

import canary

canary.directives.analyze()
canary.directives.keywords("centered_space")
canary.directives.parameterize("a,b", [(0, 5, 2), (0, 1, 2)], type=canary.centered_parameter_space)


def test():
    self = canary.get_instance()
    if self.analyze:
        return analyze()
    with open("output.json", "w") as fh:
        json.dump({"a": self.parameters.a, "b": self.parameters.b}, fh)
    return 0

will produce two test cases, one with a=1 and another with a=4, each executed in their own test directory:

$ canary describe ./centered_space/centered_space.pyt
--- centered_space ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/centered_space/centered_space.pyt
Keywords: centered_space
10 test specs using on_options=:
├── centered_space.a=0.b=0
├── centered_space.a=-10.b=0
├── centered_space.a=-5.b=0
├── centered_space.a=5.b=0
├── centered_space.a=10.b=0
├── centered_space.a=0.b=-2
├── centered_space.a=0.b=-1
├── centered_space.a=0.b=1
├── centered_space.a=0.b=2
└── centered_space
│   ├── centered_space.a=0.b=0
│   ├── centered_space.a=-10.b=0
│   ├── centered_space.a=-5.b=0
│   ├── centered_space.a=5.b=0
│   ├── centered_space.a=10.b=0
│   ├── centered_space.a=0.b=-2
│   ├── centered_space.a=0.b=-1
│   ├── centered_space.a=0.b=1
│   └── centered_space.a=0.b=2

type=random_parameter_space

def parameterize(
    names: str | tuple[str],
    values: Iterable[Iterable[str | float | int]],
    samples: int = 10,
    when: dict | str,
    type: canary.enums = canary.random_parameter_space
)

The random_parameter_space type computes random parameter values. names is a tuple or comma-separated string of parameter names and values is a list of tuples where values[i] = (start, stop) define the range from which samples random elements are taken.

Example

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.parameterize(
    "a,b", [(0, 5), (6, 10)], samples=4, type=canary.random_parameter_space
)


def test():
    self = canary.get_instance()
    print(f"a={self.parameters.a}, b={self.parameters.b}")
    return 0


if __name__ == "__main__":
    sys.exit(test())

will produce four test cases, each with a and b being chosen randomly in the range 0:5 and 6:10, respectively:

$ canary describe ./random_space/random_space.pyt
--- random_space ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/random_space/random_space.pyt
Keywords: 
4 test specs using on_options=:
├── random_space.a=4.83227.b=9.75708
├── random_space.a=2.20366.b=8.32891
├── random_space.a=0.0374574.b=8.68625
└── random_space.a=4.55488.b=6.33575

Combining multiple parameter sets

If multiple parameterize directives are issued in the same test file, the cartesian product of parameters is performed:

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.parameterize("a", (1, 4))
canary.directives.parameterize("b", (1.0e5, 1.0e6, 1.0e7))


def test():
    self = canary.test.instance
    print(f"running test with {self.parameters.a=} and {self.parameters.b=}")


if __name__ == "__main__":
    sys.exit(test())

$ canary describe parameterize/parameterize3.pyt
--- parameterize3 ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/parameterize/parameterize3.pyt
Keywords: 
6 test specs using on_options=:
├── parameterize3.a=1.b=100000
├── parameterize3.a=1.b=1e+06
├── parameterize3.a=1.b=1e+07
├── parameterize3.a=4.b=100000
├── parameterize3.a=4.b=1e+06
└── parameterize3.a=4.b=1e+07

Similarly,

# Copyright NTESS. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: MIT

import sys

import canary

canary.directives.parameterize("a,b", [(1, 1e5), (2, 1e6), (3, 1e7)])
canary.directives.parameterize("cpus", (4, 8))


def test():
    self = canary.test.instance
    print(f"running test with {self.parameters.a=}, {self.parameters.b=}, {self.parameters.cpus=}")


if __name__ == "__main__":
    sys.exit(test())

results in the following 6 test cases:

$ canary describe parameterize/parameterize4.pyt
--- parameterize4 ------------
File: /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples/parameterize/parameterize4.pyt
Keywords: 
6 test specs using on_options=:
├── parameterize4.a=1.b=100000.cpus=4
├── parameterize4.a=1.b=100000.cpus=8
├── parameterize4.a=2.b=1e+06.cpus=4
├── parameterize4.a=2.b=1e+06.cpus=8
├── parameterize4.a=3.b=1e+07.cpus=4
└── parameterize4.a=3.b=1e+07.cpus=8