Main principles for using pytest to manage tests of the CIF

0. Organization of pytest in the CIF

The scripts related to pytest are found in the bin directory and the files for defining the various test cases are in tests. In this directory, the tests are sorted according to the CTM they use.

Automatically generated tests are defined in the CIF’s directory, in .gitlab-ci.yml. They call the command tox defined in tox.ini.

1. Default tests

When a push is made on the GitLab repository, the test cases are automatically ran on the server in a so-called pipeline. Remark: since this server is not meant for heavy computing and since the tests are numerous, the test cases must be designed so that the input files and computing ressources on demand remain light-duty.

If any test fails, a message is sent to state that the pipeline has failed XXretrouver formulation exacteXX. This does not mean that the push was not taken into account XX contrairement a ce qui est dit dans la doc XXX. To check which tests have failed, the gitLab provides a dashboard accessible through menu CI/CD then Pipelines. All the details of each test are available by downloading the so-called artifacts, which include:
  • the coverage: gives the fraction of the code that is actually used by the tests. The overall value is in in coverage/.current_coverage. In reports/coverage/index.html, the coverage of all routines is summarized more in details and the html file of each routine makes it possible to check which lines are covered XXX affichage tout pourri chez moi!!!XXX

  • the standard error and output, as would have been displayed in a terminal, are available through reports/pytests-py38.html

  • the yaml files generated for running the batch of tests are stored in examples_artifact/the_CTM/ with the_CTM the name of the CTM used.

2. Run tests locally

The tests can be run locally and interactively by launching bin/pycif_test.sh: the output directory must be defined and the input path checked.

The standard error and output of the test cases are displayed in the terminal one after another. The output directory contains the usual workdir generated by the CIF for each test case.

Running tests locally can be used to check the specific features under development. It is then more efficient to run only the relevant tests. This can be done by tagging the test cases (see Add a test) and specifying conditions on (combinations of) tags, of which some examples are available in pycif_test.sh (variable mark).

3. Add a test

a) Add a new fixture

Pytest uses dictionnaries and not yaml files as input. so-called fixtures are functions which fill-in a dictionnary from which a yaml can be built for running a test.

In directory tests/the_CTM/fixtures/, __init__.py contains the refrence fixture for the_CTM. A reference fixture fills in the dictionnary for a given test case and can also perform other tasks to ensure that the test case will run. Example in tests/chimere/fixtures/__init__.py: the reference fixture is ref_chimere. In it can be found:

  • various import, including pytest and OrderedDict for dealing with the dictionary/yaml-to-be

  • the name of the fixture (here, ref_chimere) with its input arguments(s). The temporary folder where the test case is to be run, tmpdir, is mandatory. Other arguments can be added if required.

  • various paths are retrieved and/or built, to be used in the dictionnary/yaml-to-be

  • the dictionary/yam-to-be in filled in

  • the fixture yields i.e. it returns various information ( tmpdir_str, config, “short”) without terminating

  • some system actions (remove, copy, link) to ensure that the test will run correctly.

Other fixtures can be based on the reference fixture. Example in tests/chimere/fixtures/setups/melchior.py:

  • various usual import

  • the name of the fixture with its input argument, which is the name of the fixture on which it is based. Here, fixture setup_melchior is based on the reference fixture ref_chimere.

  • the basis fixture is fetched i.e. its outputs returned through yield are retrieved. The call is not exactly a usual function call but takes the form (y1, y2, y3, [...]) = name_of_the fixture).

  • the dictionary/yaml-to-be can then be accessed (here, it is config) and chosen elements can be modified.

  • do not forget to yield the required information, including the modified dictionary.

Fixtures can be based on the reference fixture or on any other available fixture. Example: tests/chimere/fixtures/setups/edgar.py is based on tests/chimere/fixtures/setups/melchior.py

It is possible to easily create multiple fixtures by using request, which makes use of params. Example in tests/chimere/fixtures/fwd.py:

  • various usual import

  • definition of the parameters which are to be changed at each request (see below). Here, 4 “setup” are created, two based on the ref_chimere fixture, two on the setup_melchior fixture. Each basis fixture is used twice: one time as is it and another time with period changed to “long”.

  • definition of (how to build a) fixture based on the parameters: its input must be request, which gives access:
    1. to the parameters as shown in the example: the name of the basis fixture is retrieved through request.param.get(“setup”, “ref_chimere”).

    2. to the output of a fixture from its name, with a different syntax from a direct call to a fixture as described above: it takes the form y1, y2, y3, [...] = request.getfixturevalue(name_of_the_fixture) as shown in the example with tmpdir_str, config, marker = request.getfixturevalue(fixture_name).

The targeted elements in the dictionnary/yaml-to-be are modified.
  • do not forget to yield the required information, including the modified dictionary

  • other actions are possible after yield. In the example, some weights are required by all fixtures in the set. Since they do not change from one to the other, being independent of the variations specified in the parameters, it is more efficient to save them when the first fixture is used so that they can be used by the other members of the set. XXXX The re-reading of the weights is not coded here but is a feature of the CIF’s pluginsXXX

b) Use the new fixture(s) in a new test

A test is a script named test_NAME_OF_THE_TEST.py (with test_ mandatory) containing function(s) named test_CASX (with test_ mandatory), whose input arguments are with fixtures. A test can be stored anywhere and is able to access the required fixtures through the import listed in conftest.py. bu default, the test makes use of the conftest.py file stored in the same directory; if none is available, the test search for a conftest.py file in the parent directory.

The main steps of a test function are generally:

  1. creating the yaml for the CIF from the dictinary/yaml-to-be provided by the fixture

  2. running the CIF with the said yaml

  3. retrieving various information regarding the outputs of the run and make assertions (assert) on them to decide whether the test was successfully run or not.

Example in tests/chimere/test_integration_1fwd.py:

  • various import which do not include any fixture

  • definitions of marks, not mandatory. Tags can be attached to a test with mark. This is useful to run only a subset of tests exploring given issues (see also Run tests locally). Here, the test is tagged as regarding CHIMERE, the forward mode, initial conditions, etc.

  • definition of a test function, named test_integration_fwd, with input argument the fixture named chimere_config_fwd. This fixture is accessible because it is imported in tests/chimere/conftest.py.

  • fetching of the fixture (with the call in the form (y1, y2, y3, [...]) = name_of_the fixture))

  • dump of the dictionary/yaml-to-be in the actual yaml file with ordered_dump

  • run of the system with this yaml file with command Setup.run_simu

  • various results are retrieved e.g. the monitor.nc file is stored in datastore monitor_ref

  • assertions defining a successfull run are made with assert. Here, for a simple forward simulation, the simulation values stored in monitor.nc must be different from NaNs for active species and equal to 0 for other species.

  • the yaml file is dumped in the directory of examples to make it easier for users who want to work with examples to build their own yaml.

It is possible to multiply tests from a basis test function, with the same general principles as is done for multiplying fixtures. For tests, this is done by defining parameters with parameterize and an input argument directing to the parameter set. All combinations of the parameters will then be used to generated as many yaml files. Example in tests/chimere/test_integration_2adjtltest.py with:

  • various import

  • definition of marks

  • definition of the parameterized sets called here settings and changing the values of parameters incrmode and testspace.

  • definition of the test function with two input arguments: the basis fixture and settings

  • fetching of the basis fixture

  • one-shot modification in the dictionary/yaml-to-be to switch from the forward mode to the adjoint mode. This could also have been achieved by defining a fixture for the adjoint mode and using it here as input. Doing this change in the test ensures that everything else is kept the same so that the adjoint test is guaranteed to match the forward configuration.

  • application of the parameterized sets for the targeted diectionary/yaml-to-be elements

  • run of the system with the yam file

  • assertion for a successful adjoint test

  • the yaml file is dumped in the directory of examples.

4. Automated local tests

For regular checks on specific machines, the tests can be run in a cron.

bin/pycif_test_for_cron.sh launch the tests and send an e-mail either to state that all have been successful or to provide the summary of the first test to fail (option -x to stop at the first fail).

Example of line to add in the crontab of the system to run the tests every 3 days at 3:14 a.m.: 14 3 */3 * * $CIF_PATH/bin/pycif_test_for_cron.sh with $CIF_PATH the path to the CIF.