Main principles for using pytest to manage tests of the CIF#
0. Organization of pytest in the CIF#
The scripts related to pytest are found in the bin
directory and the files for defining the various test cases are in tests
. In this directory, the tests are sorted according to the CTM they use.
Automatically generated tests are defined in the CIF’s directory, in .gitlab-ci.yml
. They call the command tox
defined in tox.ini
.
1. Default tests#
When a push
is made on the GitLab repository, the test cases are automatically ran on the server in a so-called pipeline. Remark: since this server is not meant for heavy computing and since the tests are numerous, the test cases must be designed so that the input files and computing ressources on demand remain light-duty.
- If any test fails, a message is sent to state that the pipeline has failed XXretrouver formulation exacteXX. This does not mean that the
push
was not taken into account XX contrairement a ce qui est dit dans la doc XXX. To check which tests have failed, the gitLab provides a dashboard accessible through menuCI/CD
thenPipelines
. All the details of each test are available by downloading the so-called artifacts, which include: the coverage: gives the fraction of the code that is actually used by the tests. The overall value is in in
coverage/.current_coverage
. Inreports/coverage/index.html
, the coverage of all routines is summarized more in details and the html file of each routine makes it possible to check which lines are covered XXX affichage tout pourri chez moi!!!XXXthe standard error and output, as would have been displayed in a terminal, are available through
reports/pytests-py38.html
the yaml files generated for running the batch of tests are stored in
examples_artifact/the_CTM/
with the_CTM the name of the CTM used.
2. Run tests locally#
The tests can be run locally and interactively by launching bin/pycif_test.sh
: the output directory must be defined and the input path checked.
The standard error and output of the test cases are displayed in the terminal one after another. The output directory contains the usual workdir generated by the CIF for each test case.
Running tests locally can be used to check the specific features under development. It is then more efficient to run only the relevant tests. This can be done by tagging the test cases (see Add a test) and specifying conditions on (combinations of) tags, of which some examples are available in pycif_test.sh (variable mark
).
3. Add a test#
a) Add a new fixture#
Pytest uses dictionnaries and not yaml files as input. so-called fixtures are functions which fill-in a dictionnary from which a yaml can be built for running a test.
In directory tests/the_CTM/fixtures/
, __init__.py
contains the refrence fixture for the_CTM. A reference fixture fills in the dictionnary for a given test case and can also perform other tasks to ensure that the test case will run.
Example in tests/chimere/fixtures/__init__.py
: the reference fixture is ref_chimere
. In it can be found:
various
import
, includingpytest
andOrderedDict
for dealing with the dictionary/yaml-to-bethe name of the fixture (here,
ref_chimere
) with its input arguments(s). The temporary folder where the test case is to be run,tmpdir
, is mandatory. Other arguments can be added if required.various paths are retrieved and/or built, to be used in the dictionnary/yaml-to-be
the dictionary/yam-to-be in filled in
the fixture yields i.e. it returns various information (
tmpdir_str, config, "short"
) without terminatingsome system actions (remove, copy, link) to ensure that the test will run correctly.
Other fixtures can be based on the reference fixture.
Example in tests/chimere/fixtures/setups/melchior.py
:
various usual
import
the name of the fixture with its input argument, which is the name of the fixture on which it is based. Here, fixture
setup_melchior
is based on the reference fixtureref_chimere
.the basis fixture is fetched i.e. its outputs returned through
yield
are retrieved. The call is not exactly a usual function call but takes the form(y1, y2, y3, [...]) = name_of_the fixture)
.the dictionary/yaml-to-be can then be accessed (here, it is
config
) and chosen elements can be modified.do not forget to yield the required information, including the modified dictionary.
Fixtures can be based on the reference fixture or on any other available fixture. Example: tests/chimere/fixtures/setups/edgar.py
is based on tests/chimere/fixtures/setups/melchior.py
It is possible to easily create multiple fixtures by using request
, which makes use of params
.
Example in tests/chimere/fixtures/fwd.py
:
various usual
import
definition of the parameters which are to be changed at each
request
(see below). Here, 4"setup"
are created, two based on theref_chimere
fixture, two on thesetup_melchior
fixture. Each basis fixture is used twice: one time as is it and another time withperiod
changed to"long"
.
- definition of (how to build a) fixture based on the parameters: its input must be
request
, which gives access:
to the parameters as shown in the example: the name of the basis fixture is retrieved through
request.param.get("setup", "ref_chimere")
.to the output of a fixture from its name, with a different syntax from a direct call to a fixture as described above: it takes the form
y1, y2, y3, [...] = request.getfixturevalue(name_of_the_fixture)
as shown in the example withtmpdir_str, config, marker = request.getfixturevalue(fixture_name)
.
- The targeted elements in the dictionnary/yaml-to-be are modified.
do not forget to yield the required information, including the modified dictionary
other actions are possible after
yield
. In the example, some weights are required by all fixtures in the set. Since they do not change from one to the other, being independent of the variations specified in the parameters, it is more efficient to save them when the first fixture is used so that they can be used by the other members of the set. XXXX The re-reading of the weights is not coded here but is a feature of the CIF’s pluginsXXX
b) Use the new fixture(s) in a new test#
A test is a script named test_NAME_OF_THE_TEST.py
(with test_
mandatory) containing function(s) named test_CASX
(with test_
mandatory), whose input arguments are with fixtures. A test can be stored anywhere and is able to access the required fixtures through the import
listed in conftest.py
. bu default, the test makes use of the conftest.py
file stored in the same directory; if none is available, the test search for a conftest.py
file in the parent directory.
The main steps of a test function are generally:
creating the yaml for the CIF from the dictinary/yaml-to-be provided by the fixture
running the CIF with the said yaml
retrieving various information regarding the outputs of the run and make assertions (
assert
) on them to decide whether the test was successfully run or not.
Example in tests/chimere/test_integration_1fwd.py
:
various
import
which do not include any fixturedefinitions of marks, not mandatory. Tags can be attached to a test with
mark
. This is useful to run only a subset of tests exploring given issues (see also Run tests locally). Here, the test is tagged as regarding CHIMERE, the forward mode, initial conditions, etc.definition of a test function, named
test_integration_fwd
, with input argument the fixture namedchimere_config_fwd
. This fixture is accessible because it is imported intests/chimere/conftest.py
.fetching of the fixture (with the call in the form
(y1, y2, y3, [...]) = name_of_the fixture)
)dump of the dictionary/yaml-to-be in the actual yaml file with
ordered_dump
run of the system with this yaml file with command
Setup.run_simu
various results are retrieved e.g. the monitor.nc file is stored in datastore
monitor_ref
assertions defining a successfull run are made with
assert
. Here, for a simple forward simulation, the simulation values stored in monitor.nc must be different from NaNs for active species and equal to 0 for other species.the yaml file is dumped in the directory of examples to make it easier for users who want to work with examples to build their own yaml.
It is possible to multiply tests from a basis test function, with the same general principles as is done for multiplying fixtures. For tests, this is done by defining parameters with parameterize
and an input argument directing to the parameter set. All combinations of the parameters will then be used to generated as many yaml files.
Example in tests/chimere/test_integration_2adjtltest.py
with:
various
import
definition of marks
definition of the parameterized sets called here
settings
and changing the values of parametersincrmode
andtestspace
.definition of the test function with two input arguments: the basis fixture and
settings
fetching of the basis fixture
one-shot modification in the dictionary/yaml-to-be to switch from the forward mode to the adjoint mode. This could also have been achieved by defining a fixture for the adjoint mode and using it here as input. Doing this change in the test ensures that everything else is kept the same so that the adjoint test is guaranteed to match the forward configuration.
application of the parameterized sets for the targeted diectionary/yaml-to-be elements
run of the system with the yam file
assertion for a successful adjoint test
the yaml file is dumped in the directory of examples.
4. Automated local tests#
For regular checks on specific machines, the tests can be run in a cron.
bin/pycif_test_for_cron.sh
launch the tests and send an e-mail either to state that all have been successful or to provide the summary of the first test to fail (option -x to stop at the first fail).
Example of line to add in the crontab of the system to run the tests every 3 days at 3:14 a.m.:
14 3 */3 * * $CIF_PATH/bin/pycif_test_for_cron.sh
with $CIF_PATH the path to the CIF.