Main CIF observation operator (standard / std)

Description

This is the main observation operator for pyCIF. It is called by most execution modes and heavily relies on so-called transforms for elementary operations.

Indeed, the observation operator can be decomposed as follows in sub-operations:

\[\mathcal{H}(\mathbf{x}) = ( \mathcal{H}_1 \circ \mathcal{H}_2 \circ \cdots \circ \mathcal{H}_N ) (\mathbf{x})\]

See details about the transforms here, in particular their individual documentation and the general input output format.

Transform pipeline

In pyCIF, the successive transforms are arranged into a so-called pipeline. The steps to initialize a pipeline consistent with the user-defined configuration are carried out in the function:

pycif.plugins.obsoperators.standard.transforms.init_transform(self)[source]

Initialize the transform pipeline according to user choices. This includes the explicit definition of sub-pipelines (main, control vector side and observation vector side), definition based on aliases in the datavect, and transforms automatically added depending on compatibility of successive input/output formats (including domain definition, dates and units).

Note

To compute a given pipeline, the observation operator first walks the pipeline backwards in a dry-run mode. This initialization step allows propagating metadata about what output format is needed for transformations.

For instance, metadata about observations need to be propagated backwards, so pyCIF knows where to extract concentrations in the CTM, before running it forward.

Main pipeline

The observation vector builds the transformation pipeline according to information specified in the control vector transform_pipe, in the observation vector transform_pipe and in the observation operator transform_pipe

The functions used to determine the main pipe are the following (by order of execution):

pycif.plugins.obsoperators.standard.transforms.init_mainpipe(self, all_transforms, backup_comps, mapper)[source]

Initialize the core of the transform_pipe depending on the list of transformations specified in obsoper.transform_pipe.

Warning

If no transform_pipe is specified, the CTM model specified in the Yaml is run by default.

On the opposite, if transform_pipe is specified in the observation operator, only transforms explicitly specified will be used. Thus, if custom transforms need to be run on top of the model, one should not forget to include the transform run_model in the transform_pipe. Another option (recommended for most applications) is to use the controlvect and obsvect transform_pipes to define transforms related to the control vector and to the observation vector respectively.

pycif.plugins.obsoperators.standard.transforms.init_control_transformations(self, all_transforms, controlvect, backup_comps, mapper)[source]

Initialize transforms on the control vector side.

Also loops on all components/tracers of the datavect and for those for which the argument unit_conversion is specified, applies the unit_conversion transform.

pycif.plugins.obsoperators.standard.transforms.init_obsvect_transformations(self, all_transforms, obsvect, backup_comps, mapper)[source]

Initialize transforms on the observation vector side.

Also, for the component satellite of the datavect, includes the transform satellites to the pipeline.

Connecting and ordering transforms into a pipeline

pycif.plugins.obsoperators.standard.transforms.connect_pipes(self, all_transforms, backup_comps, mapper)[source]

Connect transforms based on their inputs and outputs

pycif.plugins.obsoperators.standard.transforms.period_pipe(self, all_transforms, mapper)[source]

Arrange all transformations for all their sub-simulation periods into a pipe whose order respects the required precursors and successors for each transformation.

First propagate sub-simulation periods to precursors/successors for transformations which don’t have pre-defined sub-simulation periods.

Second, define a graph from all the precursors of all transformations

Last, walk the graph to define the proper order of the transformations

Parameters
  • all_transforms – the object gathering all transformations

  • mapper – the dictionary containing all information about the input/output of each transformation

Returns

the pipes to be computed in forward and backward mode, including for each direction a dry run in the other direction for initialization

Automatic pipeline

After initializing the main pipeline of required transforms, the observation operator, checks the consistency of the horizontal and vertical extent, of the temporal resolution, and of the data unit to determine extra intermediate transformations to be carried out.

More precisely, for every successive transform of the main pipeline, the observation operator checks whether the output format of the precursor transform is consistent with the input format of the successor transform. This check includes the definition of the domain (horizontal and vertical extent), of the input_dates (temporal definition) and of the unit.

The corresponding transforms that may be included at this step are:

  1. regrid

  2. time_interpolation

  3. vertical_interpolation

  4. unit_conversion

For each of the above-mentioned transforms, it is possible to explicitly specify extra parameters in the related component/tracer of the datavect as follows:

datavect :
  components:
    flux:
      parameters:
        CO2:
          dir: XXX
          file: XXX
          regrid:
            method: mass-conservation

All these operations are done in the function:

pycif.plugins.obsoperators.standard.transforms.init_default_transformations(self, all_transforms, backup_comps, mapper)[source]

Initialize default transformations based on compatibility of input/output formats of successive transforms.

Yaml arguments

The following arguments are used to configure the plugin. pyCIF will return an exception at the initialization if mandatory arguments are not specified, or if any argument does not fit accepted values or type:

Optional arguments

autorestart: (optional): False

if interrupted, computations restart from the last simulated period. WARNING: the CIF cannot detect whether this period has been correctly written or is corrupt: it is necessary to check manually in the relevant directories and remove the last simulated period if a file has not been correctly written.

accepted type: <class ‘bool’>

autoflush: (optional): False

Remove big temporary files when the run is done

accepted type: <class ‘bool’>

save_debug: (optional): False

Force transforms to save debugging information. Intermediate datastores will be saved in the directory $workdir/obsoperator/$run_id/transform_debug/

Warning

This option saves every intermediate states of the transformation pipeline. It slows drastically the computation of the obsvervation operator and can take a lot of disk space. Should be used only for debugging or understanding what happens along the way.

accepted type: <class ‘bool’>

force_full_operator: (optional): False

Force computing all transforms in the observation operator, event if no observation is to be simulated

accepted type: <class ‘bool’>

init_inputs: (optional)

Structure of components and parameters to initialize. Doing so, there is no need to define an execution mode. Only inputs that were required will be computed. Moreover, with this option, it is possible to provide a partial yaml paragraph for the datavect object: only components required to generate those required are checked before execution.

accepted structure:

any_key: (optional)

Name of a given component to be initialized

accepted structure:

parameters: (optional)

List of parameters to initialize for the corresponding component. Initialize all parameters if not specified

accepted type: <class ‘list’>

transform_pipe: (optional)

List of transformations to build the main observation operator pipeline

accepted structure:

any_key: (optional)

Name of a given transformation to be included. The name has no impact on the way the observation operator is computed, although it is recommended to use explicit names to help debugging.

accepted structure:

any_key: (optional)

Arguments to set-up the given transform

parallel: (optional)

Physical parallelization of the computation of the TL and adjoint

accepted structure:

segments: (mandatory)

Length of each parallel segment

accepted type: <class ‘str’>

overlap: (mandatory)

Length of the initial overlap with previous segments

accepted type: <class ‘str’>

ref_fwd_dir: (optional):

Path to a reference forward run. This is used when using the approximate operator to accelerate its computation.

accepted type: <class ‘str’>

approx_operator: (optional)

Approximate the observation operator outside the given interval

accepted structure:

datei: (mandatory)

Start date of the interval on which to compute the real operator

accepted type: <class ‘str’>

datef: (mandatory)

Start date of the interval on which to compute the real operator

accepted type: <class ‘str’>

batch_computation: (optional)

Compute perturbed samples of the control vector within the same observation operator

accepted structure:

nsamples: (mandatory)

Number of samples to generate

accepted type: <class ‘bool’>

ignore_model: (optional): False

Do not run the model as part of the observation operator.

accepted type: <class ‘bool’>

force_propagate_attributes: (optional): False

Force the propagation of attributes throughout transforms. Use with caution.

accepted type: <class ‘bool’>

monitor_memory: (optional): False

Print memory usage for each transform.

accepted type: <class ‘bool’>

clean_memory: (optional): True

Clean datastores that are not used anymore

accepted type: <class ‘bool’>

Requirements

The current plugin requires the present plugins to run properly:

Requirement name

Requirement type

Explicit definition

Any valid

Default name

Default version

model

Model

False

True

None

None

obsvect

ObsVect

True

True

standard

std

controlvect

ControlVect

True

True

standard

std

datavect

DataVect

True

True

standard

std

platform

Platform

True

True

None

None

Yaml template

Please find below a template for a Yaml configuration:

 1obsoperator:
 2  plugin:
 3    name: standard
 4    version: std
 5    type: obsoperator
 6
 7
 8  # Optional arguments
 9  autorestart: XXXXX
10  autoflush: XXXXX
11  save_debug: XXXXX
12  force_full_operator: XXXXX
13  init_inputs: XXXXX
14  transform_pipe: XXXXX
15  parallel: XXXXX
16  ref_fwd_dir: XXXXX
17  approx_operator: XXXXX
18  batch_computation: XXXXX
19  ignore_model: XXXXX
20  force_propagate_attributes: XXXXX
21  monitor_memory: XXXXX
22  clean_memory: XXXXX