Requirements and dependencies in pyCIF

Requirements and dependencies in pyCIF#

pyCIF automatically links plugins with each other depending on requirements specified in the source of the plugins and on elements of the configuration file.

For instance, the observation operator needs to run the numerical model at some point. For this reason, the model plugin that will be run is attached to the obsoper plugin. In the code of obsoper, the model can thus simply be called as follows:

def obsoper(self):

    # Running the model
    self.model.run(**args, **kwargs)

Another example is a model plugin that needs information about its corresponding domain plugin:

def some-model-method(self):

    # Fetching domain information
    nlon = self.domain.nlon
    nlat = self.domain.nlat

Types of dependencies#

There are two main tpes of dependencies in pyCIF:

  1. plugin A calls methods from plugin B

    For that reason, methods must have a standardized format for arguments and outputs, as specified in the corresponding pages of the documentation.

  2. plugin A needs data from plugin B

    Similarly, data format needs to follow a specified standardized format

Below is an example graph of dependencies corresponding to the pyCIF configuration file here:

digraph G { size ="4,4"; mode [shape=box, label="Mode:\nforward",style=filled,color=".7 .3 1.0"]; /*this is a comment*/ model [shape=box, label="Model:\ndummy",style=filled,color=".7 .3 1.0"]; /*this is a comment*/ controlvect [shape=box, label="Control vector:\nstandard",style=filled,color=".7 .3 1.0"]; obsoper [shape=box, label="Obs operator:\nstandard",style=filled,color=red]; obsvect [shape=box, label="Obs vector:\nstandard",style=filled,color=red]; domain [shape=box, label="Domain:\ndummy",style=filled,color=".7 .3 1.0"]; meteo [shape=box, label="Meteo:\ndummy",style=filled,color=".7 .3 1.0"]; measurements [shape=box, label="Measurements:\nrandom",style=filled,color=".7 .3 1.0"]; fluxes [shape=box, label="Fluxes:\ndummy",style=filled,color=red]; /*METHOD DEPENDENCIES*/ edge [style=bold]; mode -> obsoper; obsoper -> obsvect; obsoper -> controlvect; obsoper -> model; /*DATA DEPENDENCIES*/ edge [color=red]; obsvect -> measurements; controlvect -> model; controlvect -> domain; obsvect -> model; mode -> controlvect; obsvect -> measurements; model -> meteo; model -> domain; model -> fluxes; }

In this example, blue boxes are plugins that are explicitly defined in the configuration file. Red boxes are implicitly deduced from default values as specified in individual plugins (see here). Black arrows stand for method dependencies, while red ones are data dependencies.

Thus, for instance, the obsoper plugin is required by the mode plugin while it is not specified in the Yaml file. pyCIF automatically initializes the default dependency: obsoper (standard, std).

Definition of dependencies#

Information about requirements are specified in the __init__.py file of your plugin. To define requirements, one has to include a dictionary called requirements in the __init__.py file.

Keys of the dictionary are other plugins needed for the execution of the parent plugin. In the example below, the parent plugin will be attached a domain plugin as attribute, later callable using self.domain.

requirements = {
    "domain": {
        [...]
    }
}

Note

The possible keys in each key of the requirements dictionary are detailed below.

Please note that it is possible to give extra keys that will be used to defined arguments of the corresponding plugin.

For instance, in the case above, one can write:

requirements = {
    "domain": {
        "some_extra_key": "grub"
    }
}

Then, one will be able to call self.domain.some_extra_key in the internal functions of the corresponding plugin.

Moreover, the keys defined in that manner can be used to alter the way the domain will initialize itself.

Behaviour with dependencies#

A given plugin might need to use another plugin in different ways, which will determine how the Yaml configuration should be written and how missing requirements will be dealt with. The expected usage of dependencies is determined through the definition of each corresponding key of the dictionary requirements.

The possible arguments to provide to each key of the dictionary are the following:

  • name/version/type/subtype: name/version/type/subtype of the required plugin; type and subtype are optional if the key corresponds to a type of plugins. For instance, if the key is domain, there is no need to repeat the type.

  • any: any plugin fitting the required type is fine

  • subplug: authorizes to search for required plugins not only at the root level of the yaml.

  • preftree (to be defined if subplug si True): if there are several plugins fitting the requirement in the yaml, choose the one in the preferred tree directory. For instance, if the parent plugin requires a domain and several domain are defined, e.g., at the root level and at some sub-level of another plugin, if one of the two fits preftree, this one is selected; otherwise, an error is returned due to ambiguous choice

  • empty: an empty plugin is defined according to the name/version/type/subtype if none is available in the Yaml. This can be used when only functions of the required plugin are necessary and no data

  • newplg: force initializing a new plugin instead of looking for an existing plugin from the yaml

In practice, the requirement selection process follows the steps:

  1. pyCIF looks in the Yaml for all possible plugins fitting the parameters name/version/type/subtype/empty/subplug.

    Four possible outcomes:

    • the only one matching is returned,

    • an empty plugin is returned if there is no match,

    • if there are several matches and subplug is True: returns the one matching preftree,

    • an exception is returned if there are ambiguous choices

  2. if there is one match:

    • if any is True, attaches the fetched one

    • if any is False, attaches the fetched one if it corresponds to the specified name/version/type/subtype

    • if any is False and the fetched one does not correspond to the specified name/version/type/subtype, but empty is True, attaches an empty plugin from the default

  3. if there is no match:

    • if empty is True and any is False, attaches an empty plugin from the default

    • if empty is True and any is True, attaches an empty class plugin, with only the default class attributes

Warning

The dual options subplug/preftree should be avoided as much as possible as it means that your plugin is critically dependent on external other plugins. There are usually ways to avoid such implementation.

Cheat sheet for checking plugins dependencies in yaml files#

The following list can be obtained by running the following python commands.

from pycif.utils.classes.baseclass import Plugin
Plugin.print_registered(print_rst=True, print_requirement=True)

Available plugins#

List of all available plugins and requirements for each class:

chemistry
controlvect
  • standard, std
    Requires:
    • domain:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

datastream
datavect
  • standard, std
    Requires:
    • domain:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • components:

      • name: None

      • version: None

      • any: True

      • empty: True

domain
measurements
minimizer
mode
  • 4dvar, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • minimizer:

      • name: M1QN3

      • version: std

      • any: True

      • empty: True

    • simulator:

      • name: gausscost

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • EnSRF, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • adj-tl_test, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

  • analytic, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

  • footprint, std
    Requires:
    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: False

      • empty: True

  • forward, std
    Requires:
    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: False

      • empty: True

  • post-proc, std
    Requires:
    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

  • response-functions, std
    Requires:
    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

    • model:

      • name: None

      • version: None

      • any: True

      • empty: False

    • obsoperator:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: False

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

model
  • CHIMERE, acc
    Requires:
    • domain:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • flux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • bioflux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • meteo:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • topcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

  • CHIMERE, std
    Requires:
    • domain:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • flux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • bioflux:

      • name: CHIMERE

      • version: AEMISSIONS

      • any: False

      • empty: True

    • meteo:

      • name: CHIMERE

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • topcond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: CHIMERE

      • version: icbc

      • any: False

      • empty: True

  • ICON-ART, std

  • LMDZ, acc
    Requires:
    • domain:

      • name: LMDZ

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: LMDZ

      • version: sflx

      • any: False

      • empty: True

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • emis_species:

      • name: LMDZ

      • version: bin

      • any: False

      • empty: True

    • meteo:

      • name: LMDZ

      • version: mass-fluxes

      • any: False

      • empty: True

    • inicond:

      • name: LMDZ

      • version: ic

      • any: False

      • empty: True

    • prescrconcs:

      • name: LMDZ

      • version: prescrconcs

      • any: False

      • empty: True

    • kinetic:

      • name: LMDZ

      • version: photochem

      • any: False

      • empty: True

    • prodloss3d:

      • name: LMDZ

      • version: prodloss3d

      • any: False

      • empty: True

  • LMDZ, std
    Requires:
    • domain:

      • name: LMDZ

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: LMDZ

      • version: sflx

      • any: False

      • empty: True

    • chemistry:

      • name: CHIMERE

      • version: gasJtab

      • any: False

      • empty: False

    • emis_species:

      • name: LMDZ

      • version: bin

      • any: False

      • empty: True

    • meteo:

      • name: LMDZ

      • version: mass-fluxes

      • any: False

      • empty: True

    • inicond:

      • name: LMDZ

      • version: ic

      • any: False

      • empty: True

    • prescrconcs:

      • name: LMDZ

      • version: prescrconcs

      • any: False

      • empty: True

    • kinetic:

      • name: LMDZ

      • version: photochem

      • any: False

      • empty: True

    • prodloss3d:

      • name: LMDZ

      • version: prodloss3d

      • any: False

      • empty: True

  • Lagrangian, std
    Requires:
    • domain:

      • name: FLEXPART

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: FLEXPART

      • version: nc

      • any: False

      • empty: True

  • TM5, std
    Requires:
    • domain:

      • name: dummy

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: TM5

      • version: SINK-TIPP

      • any: False

      • empty: False

    • flux:

      • name: TM5

      • version: std

      • any: False

      • empty: True

    • meteo:

      • name: TM5

      • version: std

      • any: False

      • empty: True

    • inicond:

      • name: TM5

      • version: ic

      • any: False

      • empty: True

  • dummy, std
    Requires:
    • domain:

      • name: dummy

      • version: std

      • any: False

      • empty: False

    • flux:

      • name: dummy

      • version: nc

      • any: False

      • empty: True

    • meteo:

      • name: dummy

      • version: csv

      • any: False

      • empty: True

  • satwetch4, std

  • template, std

  • wrfchem, std
    Requires:
    • domain:

      • name: wrfchem

      • version: std

      • any: False

      • empty: False

    • chemistry:

      • name: None

      • version: None

      • any: True

      • empty: True

    • flux:

      • name: wrfchem

      • version: std

      • any: False

      • empty: True

    • latcond:

      • name: wrfchem

      • version: icbc

      • any: False

      • empty: True

    • inicond:

      • name: wrfchem

      • version: icbc

      • any: False

      • empty: True

obsoperator
  • FLEXINVERT, std
    Requires:
    • model:

      • name: None

      • version: None

      • any: True

      • empty: False

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

  • standard, std
    Requires:
    • model:

      • name: None

      • version: None

      • any: True

      • empty: False

    • obsvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • controlvect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

    • platform:

      • name: None

      • version: None

      • any: True

      • empty: True

obsparser
obsvect
  • standard, std
    Requires:
    • model:

      • name: None

      • version: None

      • any: True

      • empty: True

    • datavect:

      • name: standard

      • version: std

      • any: True

      • empty: True

platform
simulator
transform