CIF in Docker ############# .. toctree:: :hidden: gitlab-ci A docker image has been generated to easily run the CIF within a Docker container. This Docker image allows new users to try academic examples with a :doc:`Gaussian plume model`. In principle, more complex cases with full-physics numerical transport models could work on the Docker container, but this was only tested with the model :doc:`CHIMERE` so far, and it requires some extra knowledge on how to use Docker to be able to use datasets from your cluster. Docker ====== What is Docker -------------- `Docker `__ allows you to run applications inside a controlled pre-defined environment. It guarantees portability and reproducibility of your experiments. Within the CIF, Docker images are used for automatic testing, as well as for dissemination. Academic cases can be run with almost no effort for new users to get accustomed to the CIF. Installing Docker ----------------- One needs to have Docker installed on his/her machine/cluster to carry on. Please find `here `__ instructions for installing Docker. Depending on your permissions on your machines and on how Docker was installed, it may be recommended to install `Docker rootless mode `__ which enables Docker for non-root users. The CIF Docker image ==================== What is inside the CIF Docker image? ------------------------------------ The CIF Docker image is build on a Linux Ubuntu environment. It includes Python and all packages needed to run the CIF. Fortran compilers and important Fortran libraries (MPI, NetCDF) are also included to be able to run some transport models. .. note:: For developers, additional libraries can be required. You can complement accordingly the build DockerFile below to set your own Docker image. Please don't forget to share the updated image to allow other users to run your model on the Docker. .. admonition:: PyCIF ``Dockerfile`` :class: dropdown .. literalinclude:: ../../../Dockerfile :language: dockerfile Download the CIF Docker image ----------------------------- The CIF Docker images are publicly available on `Docker Hub `__. The standard image is be downloaded using the command: .. code-block:: bash docker pull pycif/pycif-ubuntu:latest .. .. warning .. Please note that image tags can change over time and may not be updated here in .. the documentation. Be sure to visit the CIF `Docker hub `__ page to .. check the last image tag. .. Please also look at the file :doc:`.gitlab-ci.yml ` .. to check what docker image is used with which test. Running the CIF inside the Docker image --------------------------------------- The CIF Docker image can be used as a container to run and test the CIF with configuration files from your machine. To run a YAML configuration file ``/home/user/my_config_file.yaml``, with a ``workdir`` in ``/scratch/user/my_workdir`` and input files in ``/data/user/`` with the CIF Docker image, you can use the following command: .. code-block:: bash docker run -v /home/user/my_config_file.yaml -v /scratch/user/ -v /data/user/ -it pycif/pycif-ubuntu:latest /home/user/my_config_file.yaml Each ``-v`` option mounts a volume from the host to the Docker container at the same path, you can specify a different path in the Docker container with ``-v /source:/dest`` (the paths in your configuration files should reflect the paths in the Docker container). The last argument is the path to the configuration file to run in the Docker container. We recommend testing tutorials for :doc:`the Gaussian plume model ` in the Docker to quick start with the CIF and explore outputs. Running automatic tests inside the Docker image ----------------------------------------------- Automatic tests are run each time a :bash:`git push` is done to `gitlab.in2p3.fr `__. It can happen that tests do not pass and that returned Exceptions are not obvious to solve. To explore tests in a dynamic way, it is possible to run the automatic tests for pyCIF manually inside a docker. To run the tests in the Docker container, you need to override the entrypoint with :bash:`tox` and pass arguments to it (here :bash:`-e 3.9 -- -m basic`). .. code-block:: bash docker run --entrypoint tox -it pycif/pycif-ubuntu:latest -e 3.9 -- -m basic The command above will run the tests in the Docker container with the version of PyCIF already installed in the container. To run the tests with the version of PyCIF you are working on, you can mount the CIF repository in the Docker container in place of the already installed version with the following command (assuming your current directory is the CIF repository): .. code-block:: bash docker run --entrypoint tox -v $(pwd)/pycif:/root/CIF/pycif -v $(pwd)/tests:/root/CIF/tests -it pycif/pycif-ubuntu:latest -e 3.9 -- -m basic For some tests, you may need to add the option :bash:`-v $PYCIF_DATA_TEST:/tmp/PYCIF_DATA_TEST/` to the command to mount the test data directory in the Docker container. It is possible to enter the docker image and explore corresponding results after running. This is especially useful if some bugs are detected. To do so: .. code-block:: bash docker ps -a # Find the ID of the container in which the tests where run docker start [container-id] # Then restart the container docker exec -it [container-id] /bin/bash # Now enter the resurrected container .. note:: External data are needed to run LMDZ, CHIMERE, FLEXPART and TM5. They can be downloaded at the following links: * `LMDZ `__, * `LMDZ Open-Acc `__, * `CHIMERE `__, * `FLEXPART `__, * `TM5 `__, * `auxiliary data `__ (e.g., EDGAR, etc; needed for CHIMERE). All external data should be extracted under a given directory that should be set as the environment variable :bash:`PYCIF_DATA_TEST` before running the tests. Further detail on `tox` commands used in CIF and about what data is downloaded for which test can be found in the file :doc:`.gitlab-ci.yml <../../../.gitlab-ci.yml>` For complicated bugs, it is possible to enter Python debugger inside :bash:`tox`. To do so, insert the following lines when you want to enter the debugger: .. code-block:: python import pudb pudb.set_trace() To allow :bash:`tox` to enter the debugger, it is necessary to add the following lines to the file :bash:`tox.ini` at the root of the CIF repository in the section :bash:`[testenv]` and the :bash:`deps=` definition (below :bash:`numpy`): .. code-block:: pudb pytest-pudb Exploring the Docker image -------------------------- You can explore the Docker image to see what is inside in an interactive shell with the following command: .. code-block:: bash docker run --entrypoint bash -it pycif/pycif-ubuntu:latest .. File structure in the image .. =========================== .. There are few tweaks to know about how to set up the path in the configuration file .. compared to running it directly on your machine. .. The working directory (:bash:`workdir`) you define in the Yaml must be valid on your host machine. .. A virtual link is established with the Docker to allow the Docker container to interact with the host. .. All the other paths should be valid from the Docker point of view. .. That means, for instance, that paths to sources in the CIF should follow the local Docker directory tree: .. :: .. Docker Container .. ├── bin .. │ └── docker_entrypoint.sh .. ├── tmp .. │ └── CIF .. │ └── CIF sources .. ├── config .. │ └── where your configuration file will be stored .. └── workdir .. └── where your case will be run (then linked back to your local workdir) .. In that case, path to CIF sources should be :bash:`/tmp/CIF/...` .. Using more complex cases with external data from the host .. ========================================================= .. If one wants to go beyond simple cases, external data (meteorological files, emission data sets, observations, etc.) .. may be necessary and not integrated in the Docker image. .. To integrate data from the host in the Docker, one needs to change the :bash:`bin/docker_debug.sh` .. and add extra volumes to the configuration. .. At line 22, the option :bash:`-v ${path_on_host}:${path_on_docker}` mounts virtual volumes in the Docker. .. Therefore, data volume can be added as follows: .. .. code-block:: bash .. docker run -v $config_file:/config/config.yml \ .. -v $pycif_root_dir:/tmp/CIF/ \ .. -v $workdir:/workdir/ \ .. -v ${path_to_my_data}:/data/ \ .. -it pycif/pycif-ubuntu:0.1 .. With that example, please remember that the paths in your configuration files pointing to your data .. will not automatically follow and would need to be updated to :bash:`/data/...`