Finding and listing test files#
canary find finds tests in a search path and prints basic information about each test.
Basic usage#
$ canary find -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
─────────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples ────────────
a alpha
analyze_only analyze_only/analyze_only.a=1.b=2.cpus=1
analyze_only/analyze_only.a=1.b=2.cpus=2 analyze_only/analyze_only.a=2.b=3.cpus=1
analyze_only/analyze_only.a=2.b=3.cpus=2 analyze_only/analyze_only.a=4.b=5.cpus=1
analyze_only/analyze_only.a=4.b=5.cpus=2 artifact
attributes beta
blt depends_on/parameter/breakfast.dish=eggs
depends_on/parameter/breakfast.dish=ham depends_on/parameter/breakfast.dish=spam
centered_space centered_space/centered_space.a=-10.b=0
centered_space/centered_space.a=-5.b=0 centered_space/centered_space.a=0.b=-1
centered_space/centered_space.a=0.b=-2 centered_space/centered_space.a=0.b=0
centered_space/centered_space.a=0.b=1 centered_space/centered_space.a=0.b=2
centered_space/centered_space.a=10.b=0 centered_space/centered_space.a=5.b=0
copy_and_link cpus
depends_on_a depends_on_willfail
diff exclusive
execute_and_analyze execute_and_analyze/execute_and_analyze.a=1
execute_and_analyze/execute_and_analyze.a=2 execute_and_analyze/execute_and_analyze.a=3
fail first
green_eggs_and_ham include
depends_on/multi/ingredients.type=bacon depends_on/multi/ingredients.type=eggs
depends_on/multi/ingredients.type=ham depends_on/multi/ingredients.type=lettuce
depends_on/multi/ingredients.type=tomato instance_attributes
kitchen_sink lunch
owners parameterize/parameterize1.a=1
parameterize/parameterize1.a=4 parameterize/parameterize2.a=1.b=2
parameterize/parameterize2.a=5.b=6 parameterize/parameterize3.a=1.b=100000
parameterize/parameterize3.a=1.b=1e+06 parameterize/parameterize3.a=1.b=1e+07
parameterize/parameterize3.a=4.b=100000 parameterize/parameterize3.a=4.b=1e+06
parameterize/parameterize3.a=4.b=1e+07 parameterize/parameterize4.a=1.b=100000.cpus=4
parameterize/parameterize4.a=1.b=100000.cpus=8 parameterize/parameterize4.a=2.b=1e+06.cpus=4
parameterize/parameterize4.a=2.b=1e+06.cpus=8 parameterize/parameterize4.a=3.b=1e+07.cpus=4
parameterize/parameterize4.a=3.b=1e+07.cpus=8 pass
random_space/random_space.a=0.0374574.b=8.68625 random_space/random_space.a=2.20366.b=8.32891
random_space/random_space.a=4.55488.b=6.33575 random_space/random_space.a=4.83227.b=9.75708
rebaseline resources/resources.cpus=1.gpus=1
resources/resources.cpus=1000.gpus=1000 resources/resources.cpus=4.gpus=4
second skip
vvt/test_exec_dir.np=3.x=1.234e6 vvt/test_exec_dir.np=4.x=1.234e7
vvt/test_exec_dir.np=5.x=1.234e8 vvt/test_exec_dir.np=6.x=1.23423476498e-2
timeout timeout
willfail xdiff
xdiff-fail xfail
xfail-code xfail-fail
Filter by keyword#
The -k option will filter tests by keyword. Eg., find tests with the basic keyword:
$ canary find -k basic -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
INFO: Selecting specs based on 1 rules
INFO: Selected 4 test specs
INFO: Excluded 86 test specs during selection
Reason Count
──────────────────────────────────────────────────
keyword expression 'basic' did not match 86
─────────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples ────────────
first second
The -k option can take a python expression, eg
$ canary find -k 'basic and second' -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
INFO: Selecting specs based on 1 rules
INFO: Selected 3 test specs
INFO: Excluded 87 test specs during selection
Reason Count
─────────────────────────────────────────────────────────────
keyword expression 'basic and second' did not match 87
─────────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples ────────────
second
Print a test DAG#
The -g option will print a graph of test dependencies:
$ canary find -g -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
├── a
├── alpha
├── analyze_only.a=1.b=2.cpus=1
├── analyze_only.a=2.b=3.cpus=1
├── analyze_only.a=4.b=5.cpus=1
├── analyze_only.a=1.b=2.cpus=2
├── analyze_only.a=2.b=3.cpus=2
├── analyze_only.a=4.b=5.cpus=2
├── artifact
├── attributes
├── beta
├── ingredients.type=bacon
├── ingredients.type=lettuce
├── ingredients.type=tomato
├── breakfast.dish=eggs
├── breakfast.dish=ham
├── breakfast.dish=spam
├── centered_space.a=0.b=0
├── centered_space.a=-10.b=0
├── centered_space.a=-5.b=0
├── centered_space.a=5.b=0
├── centered_space.a=10.b=0
├── centered_space.a=0.b=-2
├── centered_space.a=0.b=-1
├── centered_space.a=0.b=1
├── centered_space.a=0.b=2
├── copy_and_link
├── cpus
├── willfail
├── diff
├── exclusive
├── execute_and_analyze.a=1
├── execute_and_analyze.a=2
├── execute_and_analyze.a=3
├── fail
├── first
├── ingredients.type=eggs
├── ingredients.type=ham
├── include
├── instance_attributes
├── owners
├── parameterize1.a=1
├── parameterize1.a=4
├── parameterize2.a=1.b=2
├── parameterize2.a=5.b=6
├── parameterize3.a=1.b=100000
├── parameterize3.a=1.b=1e+06
├── parameterize3.a=1.b=1e+07
├── parameterize3.a=4.b=100000
├── parameterize3.a=4.b=1e+06
├── parameterize3.a=4.b=1e+07
├── parameterize4.a=1.b=100000.cpus=4
├── parameterize4.a=1.b=100000.cpus=8
├── parameterize4.a=2.b=1e+06.cpus=4
├── parameterize4.a=2.b=1e+06.cpus=8
├── parameterize4.a=3.b=1e+07.cpus=4
├── parameterize4.a=3.b=1e+07.cpus=8
├── pass
├── random_space.a=0.0374574.b=8.68625
├── random_space.a=2.20366.b=8.32891
├── random_space.a=4.55488.b=6.33575
├── random_space.a=4.83227.b=9.75708
├── rebaseline
├── resources.cpus=1.gpus=1
├── resources.cpus=1000.gpus=1000
├── resources.cpus=4.gpus=4
├── second
├── skip
├── test_exec_dir.np=3.x=1.234e6
├── test_exec_dir.np=4.x=1.234e7
├── test_exec_dir.np=5.x=1.234e8
├── test_exec_dir.np=6.x=1.23423476498e-2
├── timeout
├── timeout
├── xdiff
├── xdiff-fail
├── xfail
├── xfail-code
├── xfail-fail
├── depends_on_a
│ └── a
├── analyze_only
│ ├── analyze_only.a=1.b=2.cpus=1
│ ├── analyze_only.a=2.b=3.cpus=1
│ ├── analyze_only.a=4.b=5.cpus=1
│ ├── analyze_only.a=1.b=2.cpus=2
│ ├── analyze_only.a=2.b=3.cpus=2
│ └── analyze_only.a=4.b=5.cpus=2
├── blt
│ ├── ingredients.type=bacon
│ ├── ingredients.type=lettuce
│ └── ingredients.type=tomato
├── lunch
│ └── breakfast.dish=spam
├── centered_space
│ ├── centered_space.a=0.b=0
│ ├── centered_space.a=-10.b=0
│ ├── centered_space.a=-5.b=0
│ ├── centered_space.a=5.b=0
│ ├── centered_space.a=10.b=0
│ ├── centered_space.a=0.b=-2
│ ├── centered_space.a=0.b=-1
│ ├── centered_space.a=0.b=1
│ └── centered_space.a=0.b=2
├── depends_on_willfail
│ └── willfail
├── execute_and_analyze
│ ├── execute_and_analyze.a=1
│ ├── execute_and_analyze.a=2
│ └── execute_and_analyze.a=3
├── green_eggs_and_ham
│ ├── ingredients.type=eggs
│ └── ingredients.type=ham
└── kitchen_sink
│ ├── ingredients.type=eggs
│ ├── ingredients.type=ham
│ ├── ingredients.type=lettuce
│ ├── ingredients.type=bacon
│ └── ingredients.type=tomato
Show available keywords#
$ canary find --keywords -r .
INFO: Collecting generator files from .
INFO: Instantiating generators from collected files
INFO: Generating test specs from generators
WARNING: cmake not found, jobs cannot be generated
INFO: Searching for duplicated tests
INFO: Resolving test spec dependencies
INFO: Generated 90 test specs from 46 generators
INFO: Excluded 2 test specs during generation
Reason Count
──────────────────────────────────────────────────────────
options=enable evaluated to False for options=[] 1
intentional skipif example 1
─────────── /home/docs/checkouts/readthedocs.org/user_builds/canary-wm/checkouts/latest/src/canary/examples ────────────
basic centered_space fast include second vvt