Finding and listing test files#
canary find finds tests in a search path and prints basic information about each test.
Basic usage#
$ canary find -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
─────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples ───────
a analyze_only
analyze_only/analyze_only.a=1.b=2.cpus=1 analyze_only/analyze_only.a=1.b=2.cpus=2
analyze_only/analyze_only.a=2.b=3.cpus=1 analyze_only/analyze_only.a=2.b=3.cpus=2
analyze_only/analyze_only.a=4.b=5.cpus=1 analyze_only/analyze_only.a=4.b=5.cpus=2
blt depends_on/parameter/breakfast.dish=eggs
depends_on/parameter/breakfast.dish=ham depends_on/parameter/breakfast.dish=spam
centered_space centered_space/centered_space.a=-10.b=0
centered_space/centered_space.a=-5.b=0 centered_space/centered_space.a=0.b=-1
centered_space/centered_space.a=0.b=-2 centered_space/centered_space.a=0.b=0
centered_space/centered_space.a=0.b=1 centered_space/centered_space.a=0.b=2
centered_space/centered_space.a=10.b=0 centered_space/centered_space.a=5.b=0
copy_and_link depends_on_a
depends_on_willfail diff
execute_and_analyze execute_and_analyze/execute_and_analyze.a=1
execute_and_analyze/execute_and_analyze.a=2 execute_and_analyze/execute_and_analyze.a=3
fail first
green_eggs_and_ham depends_on/multi/ingredients.type=bacon
depends_on/multi/ingredients.type=eggs depends_on/multi/ingredients.type=ham
depends_on/multi/ingredients.type=lettuce depends_on/multi/ingredients.type=tomato
instance_attributes kitchen_sink
lunch parameterize/parameterize1.a=1
parameterize/parameterize1.a=4 parameterize/parameterize2.a=1.b=2
parameterize/parameterize2.a=5.b=6 parameterize/parameterize3.a=1.b=100000
parameterize/parameterize3.a=1.b=1e+06 parameterize/parameterize3.a=1.b=1e+07
parameterize/parameterize3.a=4.b=100000 parameterize/parameterize3.a=4.b=1e+06
parameterize/parameterize3.a=4.b=1e+07 parameterize/parameterize4.a=1.b=100000.cpus=4
parameterize/parameterize4.a=1.b=100000.cpus=8 parameterize/parameterize4.a=2.b=1e+06.cpus=4
parameterize/parameterize4.a=2.b=1e+06.cpus=8 parameterize/parameterize4.a=3.b=1e+07.cpus=4
parameterize/parameterize4.a=3.b=1e+07.cpus=8 pass
random_space/random_space.a=0.0374574.b=8.68625 random_space/random_space.a=2.20366.b=8.32891
random_space/random_space.a=4.55488.b=6.33575 random_space/random_space.a=4.83227.b=9.75708
rebaseline resources/resources.cpus=1.gpus=1
resources/resources.cpus=1000.gpus=1000 resources/resources.cpus=4.gpus=4
second skip
vvt/test_exec_dir.np=3.x=1.234e6 vvt/test_exec_dir.np=4.x=1.234e7
vvt/test_exec_dir.np=5.x=1.234e8 vvt/test_exec_dir.np=6.x=1.23423476498e-2
timeout timeout
willfail xdiff
xdiff-fail xfail
xfail-code xfail-fail
Filter by keyword#
The -k option will filter tests by keyword. Eg., find tests with the basic keyword:
$ canary find -k basic -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
INFO: Selecting specs based on 1 rules
INFO: Selected 3 test specs
INFO: Excluded 78 test specs during selection
Reason Count
──────────────────────────────────────────────────
keyword expression 'basic' did not match 78
─────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples ───────
first second
The -k option can take a python expression, eg
$ canary find -k 'basic and second' -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
INFO: Selecting specs based on 1 rules
INFO: Selected 2 test specs
INFO: Excluded 79 test specs during selection
Reason Count
─────────────────────────────────────────────────────────────
keyword expression 'basic and second' did not match 79
─────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples ───────
second
Print a test DAG#
The -g option will print a graph of test dependencies:
$ canary find -g -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
├── breakfast.dish=eggs
├── breakfast.dish=ham
├── copy_and_link
├── diff
├── fail
├── first
├── instance_attributes
├── parameterize1.a=1
├── parameterize1.a=4
├── parameterize2.a=1.b=2
├── parameterize2.a=5.b=6
├── parameterize3.a=1.b=100000
├── parameterize3.a=1.b=1e+06
├── parameterize3.a=1.b=1e+07
├── parameterize3.a=4.b=100000
├── parameterize3.a=4.b=1e+06
├── parameterize3.a=4.b=1e+07
├── parameterize4.a=1.b=100000.cpus=4
├── parameterize4.a=1.b=100000.cpus=8
├── parameterize4.a=2.b=1e+06.cpus=4
├── parameterize4.a=2.b=1e+06.cpus=8
├── parameterize4.a=3.b=1e+07.cpus=4
├── parameterize4.a=3.b=1e+07.cpus=8
├── pass
├── random_space.a=0.0374574.b=8.68625
├── random_space.a=2.20366.b=8.32891
├── random_space.a=4.55488.b=6.33575
├── random_space.a=4.83227.b=9.75708
├── rebaseline
├── resources.cpus=1.gpus=1
├── resources.cpus=1000.gpus=1000
├── resources.cpus=4.gpus=4
├── second
├── skip
├── test_exec_dir.np=3.x=1.234e6
├── test_exec_dir.np=4.x=1.234e7
├── test_exec_dir.np=5.x=1.234e8
├── test_exec_dir.np=6.x=1.23423476498e-2
├── timeout
├── timeout
├── xdiff
├── xdiff-fail
├── xfail
├── xfail-code
├── xfail-fail
├── depends_on_a
│ └── a
├── analyze_only
│ ├── analyze_only.a=1.b=2.cpus=1
│ ├── analyze_only.a=2.b=3.cpus=1
│ ├── analyze_only.a=4.b=5.cpus=1
│ ├── analyze_only.a=1.b=2.cpus=2
│ ├── analyze_only.a=2.b=3.cpus=2
│ └── analyze_only.a=4.b=5.cpus=2
├── blt
│ ├── ingredients.type=bacon
│ ├── ingredients.type=lettuce
│ └── ingredients.type=tomato
├── lunch
│ └── breakfast.dish=spam
├── centered_space
│ ├── centered_space.a=0.b=0
│ ├── centered_space.a=-10.b=0
│ ├── centered_space.a=-5.b=0
│ ├── centered_space.a=5.b=0
│ ├── centered_space.a=10.b=0
│ ├── centered_space.a=0.b=-2
│ ├── centered_space.a=0.b=-1
│ ├── centered_space.a=0.b=1
│ └── centered_space.a=0.b=2
├── depends_on_willfail
│ └── willfail
├── execute_and_analyze
│ ├── execute_and_analyze.a=1
│ ├── execute_and_analyze.a=2
│ └── execute_and_analyze.a=3
├── green_eggs_and_ham
│ ├── ingredients.type=eggs
│ └── ingredients.type=ham
└── kitchen_sink
│ ├── ingredients.type=eggs
│ ├── ingredients.type=ham
│ ├── ingredients.type=lettuce
│ ├── ingredients.type=bacon
│ └── ingredients.type=tomato
Show available keywords#
$ canary find --keywords -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, test cases cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 81 test specs from 38 generators
INFO: Excluded 1 test spec during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
─────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/release-26.4.16/src/canary/examples ───────
basic centered_space fast second