Main CIF observation operator (standard / std)

Description

This is the main observation operator for pyCIF. It is called by most execution modes and heavily relies on so-called transforms for elementary operations.

Indeed, the observation operator can be decomposed as follows in sub-operations:

\[\mathcal{H}(\mathbf{x}) = ( \mathcal{H}_1 \circ \mathcal{H}_2 \circ \cdots \circ \mathcal{H}_N ) (\mathbf{x})\]

See details about the transforms here, in particular their individual documentation and the general input output format.

Transform pipeline

In pyCIF, the successive transforms are arranged into a so-called pipeline. The steps to initialize a pipeline consistent with the user-defined configuration are carried out in the function:

pycif.plugins.obsoperators.standard.transforms.init_transform(self)[source]

Initialize the transform pipeline according to user choices. This includes the explicit definition of sub-pipelines (main, control vector side and observation vector side), definition based on aliases in the datavect, and transforms automatically added depending on compatibility of successive input/output formats (including domain definition, dates and units).

Note

To compute a given pipeline, the observation operator first walks the pipeline backwards in a dry-run mode. This initialization step allows propagating metadata about what output format is needed for transformations.

For instance, metadata about observations need to be propagated backwards, so pyCIF knows where to extract concentrations in the CTM, before running it forward.

Main pipeline

The observation vector builds the transformation pipeline according to information specified in the control vector transform_pipe, in the observation vector transform_pipe and in the observation operator transform_pipe

The functions used to determine the main pipe are the following (by order of execution):

pycif.plugins.obsoperators.standard.transforms.init_mainpipe(self, all_transforms, backup_comps, mapper)[source]

Initialize the core of the transform_pipe depending on the list of transformations specified in obsoper.transform_pipe.

Warning

If no transform_pipe is specified, the CTM model specified in the Yaml is run by default.

On the opposite, if transform_pipe is specified in the observation operator, only transforms explicitly specified will be used. Thus, if custom transforms need to be run on top of the model, one should not forget to include the transform run_model in the transform_pipe. Another option (recommended for most applications) is to use the controlvect and obsvect transform_pipes to define transforms related to the control vector and to the observation vector respectively.

pycif.plugins.obsoperators.standard.transforms.init_control_transformations(self, all_transforms, controlvect, backup_comps, mapper)[source]

Initialize transforms on the control vector side.

Also loops on all components/tracers of the datavect and for those for which the argument unit_conversion is specified, applies the unit_conversion transform.

pycif.plugins.obsoperators.standard.transforms.init_obsvect_transformations(self, all_transforms, obsvect, backup_comps, mapper)[source]

Initialize transforms on the observation vector side.

Also, for the component satellite of the datavect, includes the transform satellites to the pipeline.

Connecting and ordering transforms into a pipeline

pycif.plugins.obsoperators.standard.transforms.connect_pipes(self, all_transforms, backup_comps, mapper)[source]

Connect transforms based on their inputs and outputs

pycif.plugins.obsoperators.standard.transforms.period_pipe(self, all_transforms, mapper)[source]

Arrange all transformations for all their sub-simulation periods into a pipe whose order respects the required precursors and successors for each transformation.

First propagate sub-simulation periods to precursors/successors for transformations which don’t have pre-defined sub-simulation periods.

Second, define a graph from all the precursors of all transformations

Last, walk the graph to define the proper order of the transformations

Parameters

  • all_transforms – the object gathering all transformations

  • mapper – the dictionary containing all information about the input/output of each transformation

Returns

the pipes to be computed in forward and backward mode, including for each direction a dry run in the other direction for initialization

Automatic pipeline

After initializing the main pipeline of required transforms, the observation operator, checks the consistency of the horizontal and vertical extent, of the temporal resolution, and of the data unit to determine extra intermediate transformations to be carried out.

More precisely, for every successive transform of the main pipeline, the observation operator checks whether the output format of the precursor transform is consistent with the input format of the successor transform. This check includes the definition of the domain (horizontal and vertical extent), of the input_dates (temporal definition) and of the unit.

The corresponding transforms that may be included at this step are:

  1. regrid

  2. time_interpolation

  3. vertical_interpolation

  4. unit_conversion

For each of the above-mentioned transforms, it is possible to explicitly specify extra parameters in the related component/tracer of the datavect as follows:

datavect :
  components:
    flux:
      parameters:
        CO2:
          dir: XXX
          file: XXX
          regrid:
            method: mass-conservation

All these operations are done in the function:

pycif.plugins.obsoperators.standard.transforms.init_default_transformations(self, all_transforms, backup_comps, mapper)[source]

Initialize default transformations based on compatibility of input/output formats of successive transforms.

Yaml arguments

The following arguments are used to configure the plugin. pyCIF will return an exception at the initialization if mandatory arguments are not specified, or if any argument does not fit accepted values or type:

Optional arguments

autorestart: (optional): False

if interrupted, computations restart from the last simulated period. WARNING: the CIF cannot detect whether this period has been correctly written or is corrupt: it is necessary to check manually in the relevant directories and remove the last simulated period if a file has not been correctly written.

accepted type: bool

autoflush: (optional): False

Remove big temporary files when the run is done

accepted type: bool

save_debug: (optional): False

Force transforms to save debugging information. Intermediate datastores will be saved in the directory $workdir/obsoperator/$run_id/transform_debug/

Warning

This option saves every intermediate states of the transformation pipeline. It slows drastically the computation of the obsvervation operator and can take a lot of disk space. Should be used only for debugging or understanding what happens along the way.

accepted type: bool

force_full_operator: (optional): False

Force computing all transforms in the observation operator, event if no observation is to be simulated.

accepted type: bool

init_inputs: (optional)

Structure of components and parameters to initialize. Doing so, there is no need to define an execution mode. Only inputs that were required will be computed. Moreover, with this option, it is possible to provide a partial yaml paragraph for the datavect object: only components required to generate those required are checked before execution.

accepted structure:

any_key: (optional)

Name of a given component to be initialized

accepted structure:

parameters: (optional)

List of parameters to initialize for the corresponding component. Initialize all parameters if not specified

accepted type: list

transform_pipe: (optional)

List of transformations to build the main observation operator pipeline

accepted structure:

any_key: (optional)

Name of a given transformation to be included. The name has no impact on the way the observation operator is computed, although it is recommended to use explicit names to help debugging.

accepted structure:

**args: (optional)

Arguments to set-up the given transform

parallel: (optional)

Physical parallelization of the computation of the TL and adjoint

accepted structure:

segments: (mandatory)

Length of each parallel segment

accepted type: str

overlap: (mandatory)

Length of the initial overlap with previous segments

accepted type: str

subprocess: (optional): False

If True submit the segments in subprocesses, else submit them in new jobs with the platform plugin

accepted type: bool

nproc: (optional)

number of proc to attribute to each segments when ‘subprocess’ is True (work with LMDz only)

accepted type: int

ref_fwd_dir: (optional):

Path to a reference forward run. This is used when using the approximate operator to accelerate its computation.

accepted type: str

approx_operator: (optional)

Approximate the observation operator outside the given interval

accepted structure:

datei: (mandatory)

Start date of the interval on which to compute the real operator

accepted type: str

datef: (mandatory)

Start date of the interval on which to compute the real operator

accepted type: str

batch_computation: (optional)

Compute perturbed samples of the control vector within the same observation operator

accepted structure:

nsamples: (mandatory)

Number of samples to generate

accepted type: bool

dir_samples: (mandatory)

Directory where to fetch sample control vectors

accepted type: bool

ignore_model: (optional): False

Do not run the model as part of the observation operator.

accepted type: bool

force_propagate_attributes: (optional): False

Force the propagation of attributes throughout transforms. Use with caution.

accepted type: bool

monitor_memory: (optional): False

Print memory usage for each transform.

accepted type: bool

clean_memory: (optional): True

Clean datastores that are not used anymore

accepted type: bool

autokill_time: (optional)

Kill the running simulation after a given time and re-submit automatically. The syntax is string with a Pandas duration format, e.g., ‘23H’ to kill after 23 hours.The resource needed is to be defined in the platform section.

accepted type: str

max_resubmissions: (optional): 0

How many times the job can be automatically re-submitted

accepted type: int

Requirements

The current plugin requires the present plugins to run properly:

Requirement name

Requirement type

Explicit definition

Any valid

Default name

Default version

model

Model

False

True

None

None

obsvect

ObsVect

True

True

standard

std

controlvect

ControlVect

True

True

standard

std

datavect

DataVect

True

True

standard

std

platform

Platform

True

True

None

None

Yaml template

Please find below a template for a Yaml configuration:

 1obsoperator:
 2  plugin:
 3    name: standard
 4    version: std
 5    type: obsoperator
 6
 7  # Optional arguments
 8  autorestart: XXXXX  # bool
 9  autoflush: XXXXX  # bool
10  save_debug: XXXXX  # bool
11  force_full_operator: XXXXX  # bool
12  init_inputs:
13    any_key:
14      parameters: XXXXX  # list
15  transform_pipe:
16    any_key:
17      **args: XXXXX  # any
18  parallel:
19    segments: XXXXX  # str
20    overlap: XXXXX  # str
21    subprocess: XXXXX  # bool
22    nproc: XXXXX  # int
23  ref_fwd_dir: XXXXX  # str
24  approx_operator:
25    datei: XXXXX  # str
26    datef: XXXXX  # str
27  batch_computation:
28    nsamples: XXXXX  # bool
29    dir_samples: XXXXX  # bool
30  ignore_model: XXXXX  # bool
31  force_propagate_attributes: XXXXX  # bool
32  monitor_memory: XXXXX  # bool
33  clean_memory: XXXXX  # bool
34  autokill_time: XXXXX  # str
35  max_resubmissions: XXXXX  # int