Main principles for using pytest to manage tests of the CIF¶
0. Organization of pytest in the CIF¶
The scripts related to pytest are found in the
bin directory and the files for defining the various test cases are in
tests. In this directory, the tests are sorted according to the CTM they use.
Automatically generated tests are defined in the CIF’s directory, in
.gitlab-ci.yml. They call the command
tox defined in
1. Default tests¶
push is made on the GitLab repository, the test cases are automatically ran on the server in a so-called pipeline. Remark: since this server is not meant for heavy computing and since the tests are numerous, the test cases must be designed so that the input files and computing ressources on demand remain light-duty.
- If any test fails, a message is sent to state that the pipeline has failed XXretrouver formulation exacteXX. This does not mean that the
pushwas not taken into account XX contrairement a ce qui est dit dans la doc XXX. To check which tests have failed, the gitLab provides a dashboard accessible through menu
Pipelines. All the details of each test are available by downloading the so-called artifacts, which include:
the coverage: gives the fraction of the code that is actually used by the tests. The overall value is in in
reports/coverage/index.html, the coverage of all routines is summarized more in details and the html file of each routine makes it possible to check which lines are covered XXX affichage tout pourri chez moi!!!XXX
the standard error and output, as would have been displayed in a terminal, are available through
the yaml files generated for running the batch of tests are stored in
examples_artifact/the_CTM/with the_CTM the name of the CTM used.
2. Run tests locally¶
The tests can be run locally and interactively by launching
bin/pycif_test.sh: the output directory must be defined and the input path checked.
The standard error and output of the test cases are displayed in the terminal one after another. The output directory contains the usual workdir generated by the CIF for each test case.
Running tests locally can be used to check the specific features under development. It is then more efficient to run only the relevant tests. This can be done by tagging the test cases (see Add a test) and specifying conditions on (combinations of) tags, of which some examples are available in pycif_test.sh (variable
3. Add a test¶
a) Add a new fixture¶
Pytest uses dictionnaries and not yaml files as input. so-called fixtures are functions which fill-in a dictionnary from which a yaml can be built for running a test.
__init__.py contains the refrence fixture for the_CTM. A reference fixture fills in the dictionnary for a given test case and can also perform other tasks to ensure that the test case will run.
tests/chimere/fixtures/__init__.py: the reference fixture is
ref_chimere. In it can be found:
OrderedDictfor dealing with the dictionary/yaml-to-be
the name of the fixture (here,
ref_chimere) with its input arguments(s). The temporary folder where the test case is to be run,
tmpdir, is mandatory. Other arguments can be added if required.
various paths are retrieved and/or built, to be used in the dictionnary/yaml-to-be
the dictionary/yam-to-be in filled in
the fixture yields i.e. it returns various information (
tmpdir_str, config, “short”) without terminating
some system actions (remove, copy, link) to ensure that the test will run correctly.
Other fixtures can be based on the reference fixture.
the name of the fixture with its input argument, which is the name of the fixture on which it is based. Here, fixture
setup_melchioris based on the reference fixture
the basis fixture is fetched i.e. its outputs returned through
yieldare retrieved. The call is not exactly a usual function call but takes the form
(y1, y2, y3, [...]) = name_of_the fixture).
the dictionary/yaml-to-be can then be accessed (here, it is
config) and chosen elements can be modified.
do not forget to yield the required information, including the modified dictionary.
Fixtures can be based on the reference fixture or on any other available fixture. Example:
tests/chimere/fixtures/setups/edgar.py is based on
It is possible to easily create multiple fixtures by using
request, which makes use of
definition of the parameters which are to be changed at each
request(see below). Here, 4
“setup”are created, two based on the
ref_chimerefixture, two on the
setup_melchiorfixture. Each basis fixture is used twice: one time as is it and another time with
- definition of (how to build a) fixture based on the parameters: its input must be
request, which gives access:
to the parameters as shown in the example: the name of the basis fixture is retrieved through
to the output of a fixture from its name, with a different syntax from a direct call to a fixture as described above: it takes the form
y1, y2, y3, [...] = request.getfixturevalue(name_of_the_fixture)as shown in the example with
tmpdir_str, config, marker = request.getfixturevalue(fixture_name).
- The targeted elements in the dictionnary/yaml-to-be are modified.
do not forget to yield the required information, including the modified dictionary
other actions are possible after
yield. In the example, some weights are required by all fixtures in the set. Since they do not change from one to the other, being independent of the variations specified in the parameters, it is more efficient to save them when the first fixture is used so that they can be used by the other members of the set. XXXX The re-reading of the weights is not coded here but is a feature of the CIF’s pluginsXXX
b) Use the new fixture(s) in a new test¶
A test is a script named
test_ mandatory) containing function(s) named
test_ mandatory), whose input arguments are with fixtures. A test can be stored anywhere and is able to access the required fixtures through the
import listed in
conftest.py. bu default, the test makes use of the
conftest.py file stored in the same directory; if none is available, the test search for a
conftest.py file in the parent directory.
The main steps of a test function are generally:
creating the yaml for the CIF from the dictinary/yaml-to-be provided by the fixture
running the CIF with the said yaml
retrieving various information regarding the outputs of the run and make assertions (
assert) on them to decide whether the test was successfully run or not.
importwhich do not include any fixture
definitions of marks, not mandatory. Tags can be attached to a test with
mark. This is useful to run only a subset of tests exploring given issues (see also Run tests locally). Here, the test is tagged as regarding CHIMERE, the forward mode, initial conditions, etc.
definition of a test function, named
test_integration_fwd, with input argument the fixture named
chimere_config_fwd. This fixture is accessible because it is imported in
fetching of the fixture (with the call in the form
(y1, y2, y3, [...]) = name_of_the fixture))
dump of the dictionary/yaml-to-be in the actual yaml file with
run of the system with this yaml file with command
various results are retrieved e.g. the monitor.nc file is stored in datastore
assertions defining a successfull run are made with
assert. Here, for a simple forward simulation, the simulation values stored in monitor.nc must be different from NaNs for active species and equal to 0 for other species.
the yaml file is dumped in the directory of examples to make it easier for users who want to work with examples to build their own yaml.
It is possible to multiply tests from a basis test function, with the same general principles as is done for multiplying fixtures. For tests, this is done by defining parameters with
parameterize and an input argument directing to the parameter set. All combinations of the parameters will then be used to generated as many yaml files.
definition of marks
definition of the parameterized sets called here
settingsand changing the values of parameters
definition of the test function with two input arguments: the basis fixture and
fetching of the basis fixture
one-shot modification in the dictionary/yaml-to-be to switch from the forward mode to the adjoint mode. This could also have been achieved by defining a fixture for the adjoint mode and using it here as input. Doing this change in the test ensures that everything else is kept the same so that the adjoint test is guaranteed to match the forward configuration.
application of the parameterized sets for the targeted diectionary/yaml-to-be elements
run of the system with the yam file
assertion for a successful adjoint test
the yaml file is dumped in the directory of examples.
4. Automated local tests¶
For regular checks on specific machines, the tests can be run in a cron.
bin/pycif_test_for_cron.sh launch the tests and send an e-mail either to state that all have been successful or to provide the summary of the first test to fail (option -x to stop at the first fail).
Example of line to add in the crontab of the system to run the tests every 3 days at 3:14 a.m.:
14 3 */3 * * $CIF_PATH/bin/pycif_test_for_cron.sh
with $CIF_PATH the path to the CIF.