CIF in Docker

CIF in Docker#

A docker image has been generated to easily run the CIF within a Docker container. This Docker image allows new users to try academic examples with a Gaussian plume model. In principle, more complex cases with full-physics numerical transport models could work on the Docker container, but this was only tested with the model CHIMERE so far, and it requires some extra knowledge on how to use Docker to be able to use datasets from your cluster.

Docker#

What is Docker#

Docker allows you to run applications inside a controlled pre-defined environment. It guarantees portability and reproducibility of your experiments.

Within the CIF, Docker images are used for automatic testing, as well as for dissemination. Academic cases can be run with almost no effort for new users to get accustomed to the CIF.

Installing Docker#

One needs to have Docker installed on his/her machine/cluster to carry on. Please find here instructions for installing Docker.

Depending on your permissions on your machines and on how Docker was installed, it may be recommended to install Docker rootless mode which enables Docker for non-root users.

The CIF Docker image#

What is inside the CIF Docker image?#

The CIF Docker image is build on a Linux Ubuntu environment. It includes Python and all packages needed to run the CIF.

Fortran compilers and important Fortran libraries (MPI, NetCDF) are also included to be able to run some transport models.

Note

For developers, additional libraries can be required. You can complement accordingly the build DockerFile below to set your own Docker image. Please don’t forget to share the updated image to allow other users to run your model on the Docker.

PyCIF Dockerfile

# Build a Docker image that contains all requirements to run the PyCIF CI/CD pipeline.

FROM ubuntu:22.04

ARG HDF4_VERSION=2_16-2
ARG HDF5_VERSION=1_10_11

# Set environment variables.
ENV PY=python3.11
ENV HOME=/root 
ENV DEBIAN_FRONTEND=noninteractive
# ENV SETUPTOOLS_SCM_PRETEND_VERSION_FOR_PYCIF=0.0.1

# Install dependencies
RUN apt-get update; \
	# Basic utilities
	apt-get install --no-install-recommends -y \
		rsync git vim wget graphviz \
		python3 python3-pip python3-venv; \
	# Compilers, build utilities and libraries
	apt-get install --no-install-recommends -y \
		build-essential gcc gfortran \
		libnetcdf-dev libnetcdff-dev libhdf5-dev \
		libopenmpi-dev openmpi-bin libhdf5-openmpi-dev \
		gdal-bin libgdal-dev libproj-dev libeccodes-dev

# Install HDF4 from sources for TM5 as libhdf-dev conflicts with libgdal-dev
ADD https://github.com/HDFGroup/hdf4/archive/refs/tags/hdf-4_${HDF4_VERSION}.tar.gz /tmp/
RUN tar -xzf /tmp/hdf-4_${HDF4_VERSION}.tar.gz -C /tmp; \
	cd /tmp/hdf4-hdf-4_${HDF4_VERSION}; \
	./configure --prefix=${HOME}/.local/ --enable-fortran --disable-java \
		--disable-netcdf --enable-static --disable-shared; \
	make -j4; \
	make install; \
	rm -rf /tmp/hdf-4_${HDF4_VERSION}.tar.gz /tmp/hdf4-hdf-4_${HDF4_VERSION}/

# Install HDF5 from sources for TM5
ADD https://github.com/HDFGroup/hdf5/archive/refs/tags/hdf5-${HDF5_VERSION}.tar.gz /tmp/
RUN tar -xzf /tmp/hdf5-${HDF5_VERSION}.tar.gz -C /tmp; \
	cd /tmp/hdf5-hdf5-${HDF5_VERSION}; \
	./configure --prefix=${HOME}/.local/ --enable-fortran --disable-java \
		--enable-hl --enable-shared --enable-static; \
	make -j4; \
	make install; \
	rm -rf /tmp/hdf5-${HDF5_VERSION}.tar.gz /tmp/hdf5-hdf5-${HDF5_VERSION}/

# Install several versions of Python
RUN apt-get update; \
	apt-get install -y software-properties-common; \
	add-apt-repository -y ppa:deadsnakes/ppa; \
	apt-get install --no-install-recommends -y \
		python3.9 python3.9-venv python3.9-dev \
		python3.10 python3.10-venv python3.10-dev \
		python3.11 python3.11-venv python3.11-dev; \
	# Install tox through pip to have the latest version
	${PY} -m ensurepip --upgrade; \
	${PY} -m pip install --no-cache-dir --upgrade pip; \
	${PY} -m pip install --no-cache-dir --root-user-action=ignore tox

# Copy CIF sources
COPY ./ ${HOME}/CIF

# Set the working directory to the CIF directory
WORKDIR ${HOME}/CIF

# Install PyCIF in a virtual environment
RUN ${PY} -m venv ${HOME}/.venv; \ 
	. ${HOME}/.venv/bin/activate; \
	pip install --no-cache-dir numpy; \
	pip install --no-cache-dir --upgrade setuptools wheel; \
	pip install --no-cache-dir --no-build-isolation --force-reinstall GDAL==$(gdal-config --version); \
	pip install --no-cache-dir --editable . ; \
	rm -rf ${HOME}/.cache/pip

ENTRYPOINT [ "bin/pycif-docker" ]

Download the CIF Docker image#

The CIF Docker images are publicly available on Docker Hub. The standard image is be downloaded using the command:

docker pull pycif/pycif-ubuntu:latest

Running the CIF inside the Docker image#

The CIF Docker image can be used as a container to run and test the CIF with configuration files from your machine.

To run a YAML configuration file /home/user/my_config_file.yaml, with a workdir in /scratch/user/my_workdir and input files in /data/user/ with the CIF Docker image, you can use the following command:

docker run -v /home/user/my_config_file.yaml -v /scratch/user/ -v /data/user/ -it pycif/pycif-ubuntu:latest /home/user/my_config_file.yaml

Each -v option mounts a volume from the host to the Docker container at the same path, you can specify a different path in the Docker container with -v /source:/dest (the paths in your configuration files should reflect the paths in the Docker container). The last argument is the path to the configuration file to run in the Docker container.

We recommend testing tutorials for the Gaussian plume model in the Docker to quick start with the CIF and explore outputs.

Running automatic tests inside the Docker image#

Automatic tests are run each time a git push is done to gitlab.in2p3.fr. It can happen that tests do not pass and that returned Exceptions are not obvious to solve. To explore tests in a dynamic way, it is possible to run the automatic tests for pyCIF manually inside a docker.

To run the tests in the Docker container, you need to override the entrypoint with tox and pass arguments to it (here -e 3.9 -- -m basic).

docker run --entrypoint tox -it pycif/pycif-ubuntu:latest -e 3.9 -- -m basic

The command above will run the tests in the Docker container with the version of PyCIF already installed in the container. To run the tests with the version of PyCIF you are working on, you can mount the CIF repository in the Docker container in place of the already installed version with the following command (assuming your current directory is the CIF repository):

docker run --entrypoint tox -v $(pwd)/pycif:/root/CIF/pycif -v $(pwd)/tests:/root/CIF/tests -it pycif/pycif-ubuntu:latest -e 3.9 -- -m basic

For some tests, you may need to add the option -v $PYCIF_DATA_TEST:/tmp/PYCIF_DATA_TEST/ to the command to mount the test data directory in the Docker container.

It is possible to enter the docker image and explore corresponding results after running. This is especially useful if some bugs are detected. To do so:

docker ps -a # Find the ID of the container in which the tests where run
docker start [container-id] # Then restart the container
docker exec -it [container-id] /bin/bash # Now enter the resurrected container

Note

External data are needed to run LMDZ, CHIMERE, FLEXPART and TM5. They can be downloaded at the following links:

All external data should be extracted under a given directory that should be set as the environment variable PYCIF_DATA_TEST before running the tests.

Further detail on tox commands used in CIF and about what data is downloaded for which test can be found in the file .gitlab-ci.yml

For complicated bugs, it is possible to enter Python debugger inside tox. To do so, insert the following lines when you want to enter the debugger:

import pudb
pudb.set_trace()

To allow tox to enter the debugger, it is necessary to add the following lines to the file tox.ini at the root of the CIF repository in the section [testenv] and the deps= definition (below numpy):

pudb
pytest-pudb

Exploring the Docker image#

You can explore the Docker image to see what is inside in an interactive shell with the following command:

docker run --entrypoint bash -it pycif/pycif-ubuntu:latest
Contents