####################### Using the CIF in Docker ####################### .. role:: bash(code) :language: bash A docker image has been generated to easily run the CIF within a Docker container. This Docker image allows new users to try academic examples with a :doc:`Gaussian plume model`. In principle, more complex cases with full-physics numerical transport models could work on the Docker container, but this was only tested with the model :doc:`CHIMERE` so far, and it requires some extra knowledge on how to use Docker to be able to use datasets from your cluster. .. contents:: Contents :local: ******************* What is Docker ******************* `Docker `__ allows you to run applications inside a controlled pre-defined environment. It guarantees portability and reproducibility of your experiments. Within the CIF, Docker images are used for automatic testing, as well as for dissemination. Academic cases can be run with almost no effort for new users to get accustomed to the CIF. ****************** Installing Docker ****************** One needs to have Docker installed on his/her machine/cluster to carry on. Please find `here `__ instructions for installing Docker. Depending on your permissions on your machines and on how Docker was installed, it may be recommended to install `Docker rootless mode `__ which enables Docker for non-root users. ************************************ What is inside the CIF Docker image? ************************************ The CIF Docker image is build on a Linux Ubuntu environment. It includes Python 3.9 and all packages needed to run the CIF. Fortran compilers and important Fortran libraries (MPI, NetCDF) are also included to be able to run some transport models. .. note:: For developers, additional libraries can be required. You can complement accordingly the build DockerFile below to set your own Docker image. Please don't forget to share the updated image to allow other users to run your model on the Docker. .. literalinclude:: pycif-ubuntu.Dockerfile :language: dockerfile ********************** Download the CIF image ********************** The CIF Docker image is stored on `Docker Hub `__. The image is publicly available and can be downloaded using the command: .. code-block:: bash docker pull pycif/pycif-ubuntu:0.2 Other images are available for specific usage of the CIF, especially for some CTMs. Corresponding images can be found `here `__. .. warning:: Please note that image tags can change over time and may not be updated here in the documentation. Be sure to visit the CIF `Docker hub `__ page to check the last image tag. Please also look at the file :doc:`.gitlab-ci.yml ` to check what docker image is used with which test. ******************************** Running the CIF inside the image ******************************** The CIF Docker image can be used as a black box to run and test the CIF with configuration files from your machine. An example of script is available in the script :bash:`bin/docker_debug.sh` provided with the CIF sources. It can be simply used as follows: .. code-block:: bash cd CIF_root_dir/bin ./docker_debug.sh path_to_my_config_file.yml The script contains the following lines: .. literalinclude:: docker_debug.sh :linenos: :language: bash We recommend testing tutorials for :doc:`the Gaussian plume model ` in the Docker to quick start with the CIF and explore outputs. **************************************** Running automatic tests inside the image **************************************** Automatic tests are run each time a :bash:`git push` is done to `gitlab.in2p3.fr `__. It can happen that tests do not pass and that returned Exceptions are not obvious to solve. To explore tests in a dynamic way, it is possible to run the automatic tests for pyCIF manually inside a docker. To do so, create a script with the following lines in :bash:`${CIF_root_dir}/bin`, where :bash:`${PYCIF_DATA_TEST}` is the path to the test data (see note below for path to download test data): .. code-block:: bash #!/usr/bin/env bash # Provide local path to your yaml configuration file pycif_root_dir=`pwd`/../ extra_volumns="-v ${PYCIF_DATA_TEST}:/tmp/PYCIF_DATA_TEST/" # Run the configuration into the container and writing outputs to workdir docker run \ -v $pycif_root_dir:/tmp/CIF/ \ $extra_volumns \ -it --entrypoint /tmp/CIF/bin/tox_command.sh pycif/pycif-tm5:0.2 The script above calls another script :bash:`tox_command.sh` to be created in :bash:`${CIF_root_dir}/bin`: .. code-block:: bash #!/usr/bin/env bash cd /tmp/CIF/ pip freeze tox -e py38 -e coverage -- -m 'test_in_ci and dummy' In the above example, the tests relative to the Gaussian model will be run. It is possible to enter the docker image and explore corresponding results after running. This is especially useful if some bugs are detected. To do so: .. code-block:: bash # First find the ID of the container in which the tests where run: docker ps -a # Then restart the container docker start [container-id] # Now enter the resurrected container docker exec -it [container-id] /bin/bash .. note:: Tests are programmed for the Gaussian toy model, CHIMERE, FLEXPART and TM5. Test for TM5 should be run with the Docker image: :bash:`pycif/pycif-tm5:0.2`. Test for all other models can be run with the Docker image: :bash:`pycif/pycif-ubuntu:0.2`. The tox keywords to run one model or the other are: 1. Gaussian Toy model: .. code-block:: bash tox -e py38 -e coverage -- -m 'test_in_ci and dummy' 2. CHIMERE: .. code-block:: bash tox -e py38 -e coverage -- -m 'test_in_ci and chimere' 3. FLEXPART: .. code-block:: bash tox -e py38 -e coverage -- -m 'test_in_ci and flexpart' 4. TM5: .. code-block:: bash tox -e py38 -e coverage -- -m 'test_in_ci and tm5' External data are needed to run CHIMERE, FLEXPART and TM5. They can be downloaded at the following links: `CHIMERE `__, `FLEXPART `__, `TM5 `__, `auxiliary data `__ (e.g., EDGAR, etc; needed for CHIMERE). All external data should be extracted under a given directory that should be set as the environment variable :bash:`PYCIF_DATA_TEST` before running the tests. Further detail on `tox` commands used in CIF and about what data is downloaded for which test can be found in the file :doc:`.gitlab-ci.yml ` For complicated bugs, it is possible to enter Python debugger inside :bash:`tox`. To do so, insert the following lines when you want to enter the debugger: .. code-block:: python import pudb pudb.set_trace() To allow :bash:`tox` to enter the debugger, it is necessary to add the following lines to the file :bash:`tox.ini` at the root of the CIF repository in the section :bash:`[testenv]` and the :bash:`deps=` definition (below :bash:`numpy`): .. code-block:: pudb pytest-pudb ****************************** File structure in the image ****************************** There are few tweaks to know about how to set up the path in the configuration file compared to running it directly on your machine. The working directory (:bash:`workdir`) you define in the Yaml must be valid on your host machine. A virtual link is established with the Docker to allow the Docker container to interact with the host. All the other paths should be valid from the Docker point of view. That means, for instance, that paths to sources in the CIF should follow the local Docker directory tree: :: Docker Container ├── bin │ └── docker_entrypoint.sh ├── tmp │ └── CIF │ └── CIF sources ├── config │ └── where your configuration file will be stored └── workdir └── where your case will be run (then linked back to your local workdir) In that case, path to CIF sources should be :bash:`/tmp/CIF/...` ********************************************************* Using more complex cases with external data from the host ********************************************************* If one wants to go beyond simple cases, external data (meteorological files, emission data sets, observations, etc.) may be necessary and not integrated in the Docker image. To integrate data from the host in the Docker, one needs to change the :bash:`bin/docker_debug.sh` and add extra volumes to the configuration. At line 22, the option :bash:`-v ${path_on_host}:${path_on_docker}` mounts virtual volumes in the Docker. Therefore, data volume can be added as follows: .. code-block:: bash docker run -v $config_file:/config/config.yml \ -v $pycif_root_dir:/tmp/CIF/ \ -v $workdir:/workdir/ \ -v ${path_to_my_data}:/data/ \ -it pycif/pycif-ubuntu:0.1 With that example, please remember that the paths in your configuration files pointing to your data will not automatically follow and would need to be updated to :bash:`/data/...`