######################################################################################################## Main principles for using pytest to manage tests of the CIF ######################################################################################################## .. role:: bash(code) :language: bash 0. Organization of pytest in the CIF ------------------------------------- The scripts related to pytest are found in the :bash:`bin` directory and the files for defining the various test cases are in :bash:`tests`. In this directory, the tests are sorted according to the CTM they use. Automatically generated tests are defined in the CIF's directory, in :bash:`.gitlab-ci.yml`. They call the command :bash:`tox` defined in :bash:`tox.ini`. 1. Default tests ----------------- When a :bash:`push` is made on the GitLab repository, the test cases are automatically ran on the server in a so-called pipeline. **Remark:** since this server is not meant for heavy computing and since the tests are numerous, the test cases must be designed so that the input files and computing ressources on demand remain light-duty. If any test fails, a message is sent to state that the pipeline has failed XXretrouver formulation exacteXX. This does not mean that the :bash:`push` was not taken into account XX contrairement a ce qui est dit dans la doc XXX. To check which tests have failed, the gitLab provides a dashboard accessible through menu :bash:`CI/CD` then :bash:`Pipelines`. All the details of each test are available by downloading the so-called artifacts, which include: - the coverage: gives the fraction of the code that is actually used by the tests. The overall value is in in :bash:`coverage/.current_coverage`. In :bash:`reports/coverage/index.html`, the coverage of all routines is summarized more in details and the html file of each routine makes it possible to check which lines are covered XXX affichage tout pourri chez moi!!!XXX - the standard error and output, as would have been displayed in a terminal, are available through :bash:`reports/pytests-py38.html` - the yaml files generated for running the batch of tests are stored in :bash:`examples_artifact/the_CTM/` with the_CTM the name of the CTM used. 2. Run tests locally --------------------- The tests can be run locally and interactively by launching :bash:`bin/pycif_test.sh`: the output directory must be defined and the input path checked. The standard error and output of the test cases are displayed in the terminal one after another. The output directory contains the usual workdir generated by the CIF for each test case. Running tests locally can be used to check the specific features under development. It is then more efficient to run only the relevant tests. This can be done by tagging the test cases (see Add a test) and specifying conditions on (combinations of) tags, of which some examples are available in pycif_test.sh (variable :bash:`mark`). 3. Add a test -------------- a) Add a new fixture ******************** Pytest uses dictionnaries and not yaml files as input. so-called fixtures are functions which fill-in a dictionnary from which a yaml can be built for running a test. In directory :bash:`tests/the_CTM/fixtures/`, :bash:`__init__.py` contains the refrence fixture for the_CTM. A reference fixture fills in the dictionnary for a given test case and can also perform other tasks to ensure that the test case will run. Example in :bash:`tests/chimere/fixtures/__init__.py`: the reference fixture is :bash:`ref_chimere`. In it can be found: - various :bash:`import`, including :bash:`pytest` and :bash:`OrderedDict` for dealing with the dictionary/yaml-to-be - the name of the fixture (here, :bash:`ref_chimere`) with its input arguments(s). The temporary folder where the test case is to be run, :bash:`tmpdir`, is mandatory. Other arguments can be added if required. - various paths are retrieved and/or built, to be used in the dictionnary/yaml-to-be - the dictionary/yam-to-be in filled in - the fixture yields i.e. it returns various information ( :bash:`tmpdir_str, config, "short"`) **without** terminating - some system actions (remove, copy, link) to ensure that the test will run correctly. Other fixtures can be based on the reference fixture. Example in :bash:`tests/chimere/fixtures/setups/melchior.py`: - various usual :bash:`import` - the name of the fixture with its input argument, which is the name of the fixture on which it is based. Here, fixture :bash:`setup_melchior` is based on the reference fixture :bash:`ref_chimere`. - the basis fixture is fetched i.e. its outputs returned through :bash:`yield` are retrieved. The call is not exactly a usual function call but takes the form :bash:`(y1, y2, y3, [...]) = name_of_the fixture)`. - the dictionary/yaml-to-be can then be accessed (here, it is :bash:`config`) and chosen elements can be modified. - do not forget to yield the required information, including the modified dictionary. Fixtures can be based on the reference fixture or on any other available fixture. Example: :bash:`tests/chimere/fixtures/setups/edgar.py` is based on :bash:`tests/chimere/fixtures/setups/melchior.py` It is possible to easily create multiple fixtures by using :bash:`request`, which makes use of :bash:`params`. Example in :bash:`tests/chimere/fixtures/fwd.py`: - various usual :bash:`import` - definition of the parameters which are to be changed at each :bash:`request` (see below). Here, 4 :bash:`"setup"` are created, two based on the :bash:`ref_chimere` fixture, two on the :bash:`setup_melchior` fixture. Each basis fixture is used twice: one time as is it and another time with :bash:`period` changed to :bash:`"long"`. - definition of (how to build a) fixture based on the parameters: its input must be :bash:`request`, which gives access: i) to the parameters as shown in the example: the name of the basis fixture is retrieved through :bash:`request.param.get("setup", "ref_chimere")`. ii) to the output of a fixture from its name, with a different syntax from a direct call to a fixture as described above: it takes the form :bash:`y1, y2, y3, [...] = request.getfixturevalue(name_of_the_fixture)` as shown in the example with :bash:`tmpdir_str, config, marker = request.getfixturevalue(fixture_name)`. The targeted elements in the dictionnary/yaml-to-be are modified. - do not forget to yield the required information, including the modified dictionary - other actions are possible after :bash:`yield`. In the example, some weights are required by all fixtures in the set. Since they do not change from one to the other, being independent of the variations specified in the parameters, it is more efficient to save them when the first fixture is used so that they can be used by the other members of the set. XXXX The re-reading of the weights is not coded here but is a feature of the CIF's pluginsXXX b) Use the new fixture(s) in a new test ---------------------------------------- A test is a script named :bash:`test_NAME_OF_THE_TEST.py` (with :bash:`test_` mandatory) containing function(s) named :bash:`test_CASX` (with :bash:`test_` mandatory), whose input arguments are with fixtures. A test can be stored anywhere and is able to access the required fixtures through the :bash:`import` listed in :bash:`conftest.py`. bu default, the test makes use of the :bash:`conftest.py` file stored in the same directory; if none is available, the test search for a :bash:`conftest.py` file in the parent directory. The main steps of a test function are generally: 1) creating the yaml for the CIF from the dictinary/yaml-to-be provided by the fixture 2) running the CIF with the said yaml 3) retrieving various information regarding the outputs of the run and make assertions (:bash:`assert`) on them to decide whether the test was successfully run or not. Example in :bash:`tests/chimere/test_integration_1fwd.py`: - various :bash:`import` which do not include any fixture - definitions of marks, not mandatory. Tags can be attached to a test with :bash:`mark`. This is useful to run only a subset of tests exploring given issues (see also Run tests locally). Here, the test is tagged as regarding CHIMERE, the forward mode, initial conditions, etc. - definition of a test function, named :bash:`test_integration_fwd`, with input argument the fixture named :bash:`chimere_config_fwd`. This fixture is accessible because it is imported in :bash:`tests/chimere/conftest.py`. - fetching of the fixture (with the call in the form :bash:`(y1, y2, y3, [...]) = name_of_the fixture)`) - dump of the dictionary/yaml-to-be in the actual yaml file with :bash:`ordered_dump` - run of the system with this yaml file with command :bash:`Setup.run_simu` - various results are retrieved e.g. the monitor.nc file is stored in datastore :bash:`monitor_ref` - assertions defining a successfull run are made with :bash:`assert`. Here, for a simple forward simulation, the simulation values stored in monitor.nc must be different from NaNs for active species and equal to 0 for other species. - the yaml file is dumped in the directory of examples to make it easier for users who want to work with examples to build their own yaml. It is possible to **multiply** tests from a basis test function, with the same general principles as is done for multiplying fixtures. For tests, this is done by defining parameters with :bash:`parameterize` and an input argument directing to the parameter set. All combinations of the parameters will then be used to generated as many yaml files. Example in :bash:`tests/chimere/test_integration_2adjtltest.py` with: - various :bash:`import` - definition of marks - definition of the parameterized sets called here :bash:`settings` and changing the values of parameters :bash:`incrmode` and :bash:`testspace`. - definition of the test function with two input arguments: the basis fixture and :bash:`settings` - fetching of the basis fixture - one-shot modification in the dictionary/yaml-to-be to switch from the forward mode to the adjoint mode. This could also have been achieved by defining a fixture for the adjoint mode and using it here as input. Doing this change in the test ensures that everything else is kept the same so that the adjoint test is guaranteed to match the forward configuration. - application of the parameterized sets for the targeted diectionary/yaml-to-be elements - run of the system with the yam file - assertion for a successful adjoint test - the yaml file is dumped in the directory of examples. 4. Automated local tests ------------------------- For regular checks on specific machines, the tests can be run in a cron. :bash:`bin/pycif_test_for_cron.sh` launch the tests and send an e-mail either to state that all have been successful or to provide the summary of the first test to fail (option -x to stop at the first fail). Example of line to add in the crontab of the system to run the tests every 3 days at 3:14 a.m.: :bash:`14 3 */3 * * $CIF_PATH/bin/pycif_test_for_cron.sh` with $CIF_PATH the path to the CIF.