Requirements and dependencies in pyCIF

Requirements and dependencies in pyCIF#

pyCIF automatically links plugins with each other depending on requirements specified in the source of the plugins and on elements of the configuration file.

For instance, the observation operator needs to run the numerical model at some point. For this reason, the model plugin that will be run is attached to the obsoper plugin. In the code of obsoper, the model can thus simply be called as follows:

def obsoper(self):

    # Running the model
    self.model.run(**args, **kwargs)

Another example is a model plugin that needs information about its corresponding domain plugin:

def some-model-method(self):

    # Fetching domain information
    nlon = self.domain.nlon
    nlat = self.domain.nlat

Types of dependencies#

There are two main types of dependencies in pyCIF:

  1. plugin A calls methods from plugin B

    For that reason, methods must have a standardized format for arguments and outputs, as specified in the corresponding pages of the documentation.

  2. plugin A needs data from plugin B

    Similarly, data format needs to follow a specified standardized format

Below is an example graph of dependencies corresponding to the pyCIF configuration file here:

../../_images/dependencies.svg

In this example, blue boxes are plugins that are explicitly defined in the configuration file. Red boxes are implicitly deduced from default values as specified in individual plugins (see here). Black arrows stand for method dependencies, while red ones are data dependencies.

Thus, for instance, the obsoper plugin is required by the mode plugin while it is not specified in the Yaml file. pyCIF automatically initializes the default dependency: obsoper (standard, std).

Definition of dependencies#

Information about requirements are specified in the __init__.py file of your plugin. To define requirements, one has to include a dictionary called requirements in the __init__.py file.

Keys of the dictionary are other plugins needed for the execution of the parent plugin. In the example below, the parent plugin will be attached a domain plugin as attribute, later callable using self.domain.

requirements = {
    "domain": {
        [...]
    }
}

Note

The possible keys in each key of the requirements dictionary are detailed below.

Please note that it is possible to give extra keys that will be used to define arguments of the corresponding plugin.

For instance, in the case above, one can write:

requirements = {
    "domain": {
        "some_extra_key": "grub"
    }
}

Then, one will be able to call self.domain.some_extra_key in the internal functions of the corresponding plugin.

Moreover, the keys defined in that manner can be used to alter the way the domain will initialize itself.

Warning

The following keys are reserved by the dependency mechanism and are consumed internally. They will not be forwarded as attributes to the attached plugin, so they cannot be used as custom extra keys:

name, version, type, any, subplug, preftree, empty, newplg

Behaviour with dependencies#

A given plugin might need to use another plugin in different ways, which will determine how the Yaml configuration should be written and how missing requirements will be dealt with. The expected usage of dependencies is determined through the definition of each corresponding key of the dictionary requirements.

The possible arguments to provide to each key of the dictionary are the following:

  • name/version/type/subtype: name/version/type/subtype of the required plugin; type and subtype are optional if the key corresponds to a type of plugins. For instance, if the key is domain, there is no need to repeat the type.

  • any: any plugin fitting the required type is fine

  • subplug: authorizes to search for required plugins not only at the root level of the yaml.

  • preftree (to be defined if subplug is True): when several plugins fit the requirement, select the one whose YAML tree path contains the preftree substring. Among matching candidates, the one at the shallowest level (shortest path) is chosen. If no candidate matches preftree, pyCIF falls through to the empty fallback rather than raising an error.

  • empty: an empty plugin is defined according to the name/version/type/subtype if none is available in the Yaml. This can be used when only functions of the required plugin are necessary and no data.

  • newplg: force initializing a new plugin instance instead of reusing the existing one from the Yaml.

The resolution of each requirement is implemented in _fetch_requirement() (see the API reference). In practice, it follows a strict priority chain (first match wins):

  1. Child attribute — a plugin explicitly defined as a child of the parent plugin in the Yaml (i.e., a sub-paragraph of the parent’s Yaml block).

  2. Level-0 reference — a plugin defined at the root level of the Yaml file (i.e., a top-level paragraph such as model, domain, etc.).

  3. Unambiguous sub-reference (subplug: True only) — if there is exactly one plugin of the required type anywhere in the Yaml tree, use it.

  4. Disambiguated sub-reference (subplug: True only) — if several candidates exist, select the one whose Yaml path best matches preftree (shallowest path wins).

  5. Empty fallback (empty: True) — load the registered default plugin of the required name/version/type, or instantiate a bare subclass if none is registered.

  6. Error — if none of the above conditions can be satisfied, a PluginError is raised.

The diagram below summarises this priority chain:

Flowchart of Plugin._fetch_requirement() priority chain

Note

Steps 1 and 2 establish a precedence: a child plugin defined inline under the parent plugin in the Yaml shadows a top-level plugin of the same type. This matters when, for example, a model defines its own domain that differs from a domain defined at the root level.

Once a candidate is found, pyCIF decides whether to attach it directly or replace it with the registered default:

  • If any is True and the candidate has a valid plugin identity, attach it as-is.

  • If any is False, attach it only if its name/version/type/subtype match the requirement exactly.

  • If any is False and the candidate does not match, but empty is True, attach an empty plugin from the registered default instead.

  • If empty is True and any is True but no named candidate was found, attach a bare empty class instance with only default class attributes.

Warning

The dual options subplug/preftree should be avoided as much as possible as it means that your plugin is critically dependent on other external plugins. There are usually ways to avoid such implementation.

Dynamic requirements with set_requirements#

Before looping over the requirements dictionary, pyCIF calls the method set_requirements() on the plugin instance. Plugins can override this method to modify their own requirements at runtime, for example to activate or skip a dependency based on a Yaml parameter already loaded:

def set_requirements(self):
    # Add a chemistry plugin only when the model has species
    if getattr(self, "use_chemistry", False):
        self.requirements["chemistry"] = {
            "name": "standard", "version": "std", "empty": True
        }

The base implementation of set_requirements does nothing. This hook is intended for advanced plugin developers who need conditional dependencies.

Cheat sheet for checking plugins dependencies in yaml files#

The following list can be obtained by running the following python commands.

from pycif.utils.classes.baseclass import Plugin
Plugin.print_registered(print_rst=True, print_requirement=True)

Available plugins#

List of all available plugins and requirements for each class:

chemistry
controlvect
  • standard, std
    Requires:
    • domain:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

datastream
datavect
  • standard, std
    Requires:
    • domain:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • components:

      • name: None

      • version: None

      • any: True

      • empty: True

domain
measurements
minimizer
mode
  • 4dvar, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • minimizer:

      • name: M1QN3

      • version: std

      • any: True

      • empty: True

    • simulator:

      • name: gausscost

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • EnSRF, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • adj-tl_test, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • simulator:

      • name: gausscost

      • version: std

      • any: True

      • empty: True

  • analytic, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • co2mvs, std
    Requires:
    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: False

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • footprint, std
    Requires:
    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: False

      • empty: True

  • forward, std
    Requires:
    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: False

      • empty: True

  • post-proc, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

  • response-functions, std
    Requires:
    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: False

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

model
  • CHIMERE, acc
    Requires:
    • domain:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • flux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • bioflux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • meteo:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • topcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

  • CHIMERE, std
    Requires:
    • domain:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • flux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • bioflux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • meteo:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • topcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

  • ICON-ART, std

  • LMDZ, acc
    Requires:
    • domain:

      • name: LMDZ

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: LMDZ

      • version: sflx

      • any: False

      • empty: True

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • emis_species:

      • name: LMDZ

      • version: bin

      • any: False

      • empty: True

    • meteo:

      • name: LMDZ

      • version: mass-fluxes-acc

      • any: False

      • empty: True

    • inicond:

      • name: LMDZ

      • version: ic

      • any: False

      • empty: True

    • prescrconcs:

      • name: LMDZ

      • version: prescrconcs

      • any: False

      • empty: True

    • kinetic:

      • name: LMDZ

      • version: photochem

      • any: False

      • empty: True

    • prodloss3d:

      • name: LMDZ

      • version: prodloss3d

      • any: False

      • empty: True

  • LMDZ, reg-ico

  • LMDZ, std
    Requires:
    • domain:

      • name: LMDZ

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: LMDZ

      • version: sflx

      • any: False

      • empty: True

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • emis_species:

      • name: LMDZ

      • version: bin

      • any: False

      • empty: True

    • meteo:

      • name: LMDZ

      • version: mass-fluxes

      • any: False

      • empty: True

    • inicond:

      • name: LMDZ

      • version: ic

      • any: False

      • empty: True

    • prescrconcs:

      • name: LMDZ

      • version: prescrconcs

      • any: False

      • empty: True

    • kinetic:

      • name: LMDZ

      • version: photochem

      • any: False

      • empty: True

    • prodloss3d:

      • name: LMDZ

      • version: prodloss3d

      • any: False

      • empty: True

  • Lagrangian, std
    Requires:
    • domain:

      • name: FLEXPART

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: FLEXPART

      • version: nc

      • any: False

      • empty: True

  • TM5, std
    Requires:
    • domain:

      • name: dummy

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: TM5

      • version: SINK-TIPP

      • any: False

      • empty: False

    • flux:

      • name: TM5

      • version: std

      • any: False

      • empty: True

    • meteo:

      • name: TM5

      • version: std

      • any: False

      • empty: True

    • inicond:

      • name: TM5

      • version: ic

      • any: False

      • empty: True

  • dummy, std
    Requires:
    • domain:

      • name: dummy

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: dummy

      • version: nc

      • any: False

      • empty: True

    • meteo:

      • name: dummy

      • version: csv

      • any: False

      • empty: True

  • satwetch4, std

  • template, std

  • wrfchem, std
    Requires:
    • domain:

      • name: wrfchem

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: None

      • version: None

      • any: True

      • empty: True

    • flux:

      • name: wrfchem

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: wrfchem

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: wrfchem

      • version: icbc

      • any: False

      • empty: True

obsoperator
  • random, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • standard, std
    Requires:
    • model:

      • name: None

      • version: None

      • any: True

      • empty: False

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

obsparser
obsvect
  • standard, std
    Requires:
    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

platform
simulator
transform