Main CIF observation operator standard/std

Main CIF observation operator standard/std#

Description#

This is the main observation operator for pyCIF. It is called by most execution modes and heavily relies on so-called transforms for elementary operations.

Indeed, the observation operator can be decomposed as follows in sub-operations:

\[\mathcal{H}(\mathbf{x}) = ( \mathcal{H}_1 \circ \mathcal{H}_2 \circ \cdots \circ \mathcal{H}_N ) (\mathbf{x})\]

See details about the transforms here, in particular their individual documentation and the general input output format.

Transform pipeline#

In pyCIF, the successive transforms are arranged into a so-called pipeline. The steps to initialize a pipeline consistent with the user-defined configuration are carried out in the function:

pycif.plugins.obsoperators.standard.transforms.init_transform(self)[source]

Initialize the complete transform pipeline for the observation operator.

Assembles the ordered Transform pipeline that the operator will execute at run time. The pipeline is built from four sub-pipelines applied in sequence:

  1. Observation-vector side — transforms from obsvect.transform_pipe plus a mandatory toobsvect step for each observed species, and a satellites step for satellite components (via init_obsvect_transformations()).

  2. Main pipe — transforms from obsoperator.transform_pipe (defaults to a single run_model step when none are specified), via init_mainpipe().

  3. Control-vector side — transforms from controlvect.transform_pipe (via init_control_transformations()).

  4. Dump / load wrappers — dump2inputs and loadfromoutputs transforms inserted automatically where force_dump or force_loadout flags are set (via dump_read_inout()).

After assembly the pipeline is ordered so that precursor transforms always run before their successors via period_pipe(). Data availability is verified by check_datavect(), and a human-readable description is written to disk by dump_transform_description().

If self.batch_computation is configured, the pipeline is further modified for Monte-Carlo batch execution via batch_computation().

Parameters:

self (ObsOperator) – the obs-operator plugin instance. On return, self.transform_pipe, self.period_order_fwd, and self.period_order_adj are populated.

Note

To compute a given pipeline, the observation operator first walks the pipeline backwards in a dry-run mode. This initialization step allows propagating metadata about what output format is needed for transformations.

For instance, metadata about observations need to be propagated backwards, so pyCIF knows where to extract concentrations in the CTM, before running it forward.

Main pipeline#

The observation vector builds the transformation pipeline according to information specified in the control vector transform_pipe, in the observation vector transform_pipe and in the observation operator transform_pipe

The functions used to determine the main pipe are the following (by order of execution):

pycif.plugins.obsoperators.standard.transforms.init_mainpipe(self, all_transforms, backup_comps, mapper)[source]

Initialize the core of the transform pipeline.

Reads self.transform_pipe (transforms defined directly on the observation operator in the YAML) and inserts each of its transforms before the first element already present in self.mainpipe. If no transform_pipe is defined and self.ignore_model is False, a default run_model transform is added automatically.

Warning

If transform_pipe is specified in the observation operator, only the explicitly listed transforms are used — the CTM model is not added automatically. To run the model on top of custom transforms, include run_model explicitly in the list. For most applications it is preferable to define extra transforms in the controlvect or obsvect transform_pipe instead.

Parameters:

  • self (ObsOperator) – the obs-operator plugin instance. On return, self.mainpipe is updated with the IDs of the newly inserted transforms.

  • all_transforms – the Transform object holding all transforms; modified in-place.

  • backup_comps (dict) – backed-up component definitions forwarded to add_default().

  • mapper (dict) – the pipeline mapper dictionary; updated in-place.

pycif.plugins.obsoperators.standard.transforms.init_control_transformations(self, all_transforms, controlvect, backup_comps, mapper)[source]

Initialize transforms on the control-vector side.

Reads controlvect.transform_pipe and inserts each of its transforms before the first element of self.mainpipe in all_transforms, preserving the user-defined order.

Also loops over all components/tracers of the datavect and, for those that specify the unit_conversion argument, automatically inserts a unit_conversion transform.

Parameters:

  • self (ObsOperator) – the obs-operator plugin instance; uses self.mainpipe to determine the insertion point.

  • all_transforms – the Transform object holding all transforms; modified in-place.

  • controlvect (ControlVect) – control-vector object; its transform_pipe is read to determine which transforms to insert.

  • backup_comps (dict) – backed-up component definitions forwarded to add_default().

  • mapper (dict) – the pipeline mapper dictionary; updated in-place.

pycif.plugins.obsoperators.standard.transforms.init_obsvect_transformations(self, all_transforms, obsvect, backup_comps, mapper)[source]

Initialize transforms on the observation-vector side.

Appends a toobsvect transform to the pipeline for every observed species (component/tracer pair where param.isobs is True). For satellites components, a satellites transform is inserted immediately before the corresponding toobsvect step.

Then reads obsvect.transform_pipe and prepends each of its transforms before all other transforms in all_transforms, preserving the user-defined order.

Parameters:

  • self (ObsOperator) – the obs-operator plugin instance. On return, self.mainpipe is populated with the IDs of the newly inserted toobsvect (and satellites) transforms.

  • all_transforms – the Transform object holding all transforms; modified in-place.

  • obsvect (ObsVect) – observation-vector object; its transform_pipe and datavect are read to determine which transforms to insert.

  • backup_comps (dict) – backed-up component definitions forwarded to add_default().

  • mapper (dict) – the pipeline mapper dictionary; updated in-place.

Connecting and ordering transforms into a pipeline#

pycif.plugins.obsoperators.standard.transforms.connect_pipes(all_transforms, mapper, transform)[source]

Wire a newly added transform into the pipe based on shared trids.

For each of transform’s input trids, scans every transform registered before it for a matching output trid and records it as a precursor (symmetrically, successors are found by scanning transforms registered after it). Pre-existing precursors/successors (e.g. forced via add_default.update_successors_precursors()) are left untouched.

Then prunes redundant indirect links: if a precursor of transform for a trid is itself reachable through another of that trid’s precursors, it is removed from the direct list (and symmetrically for successors), so that only the most immediate producer/consumer chain is kept. If several precursors remain for the same trid on a downstream transform, only the closest one (last in pipe order) is kept, and the corresponding transforms’ successors lists are updated to match.

Parameters:

  • all_transforms – Namespace holding all registered transform instances, in pipe order, used to scan for candidate precursors/successors.

  • mapper – Dictionary mapping each transform id to its inputs/outputs/ precursors/successors metadata, mutated in place.

  • transform – Id of the transform to connect to the rest of the pipe.

Raises:

  • CifAttributeError – If transform is not yet registered in

  • all_transforms.attributes` –

pycif.plugins.obsoperators.standard.transforms.period_pipe(self, all_transforms, mapper)[source]

Arrange all transforms into ordered forward and adjoint execution pipes.

Determines the chronologically correct execution order for every (transform, sub-simulation date) pair by:

  1. Propagating sub-simulation periods from each transform to its precursors and successors via default_subsimus().

  2. Building a dependency graph and walking it in forward order with fwd_adj_pipe() (mode='forward').

  3. Walking the same graph in reverse order (mode='adjoint').

Each returned pipe is a list of (date, transform_id, direction) tuples, where direction is either 'forward' or 'adjoint' and controls whether a transform runs in its normal or dry-run mode.

Parameters:

  • self (ObsOperator) – the obs-operator plugin instance.

  • all_transforms – the Transform object holding all initialized transforms.

  • mapper (dict) – the pipeline mapper dictionary mapping transform IDs to their sub-simulation, input/output and precursor/successor metadata.

Returns:

(pipe_fwd, pipe_adj) where each element is a list of (datetime.datetime, str, str) tuples giving the execution order for forward and adjoint runs respectively.

Return type:

tuple[list, list]

Automatic pipeline#

After initializing the main pipeline of required transforms, the observation operator, checks the consistency of the horizontal and vertical extent, of the temporal resolution, and of the data unit to determine extra intermediate transformations to be carried out.

More precisely, for every successive transform of the main pipeline, the observation operator checks whether the output format of the precursor transform is consistent with the input format of the successor transform. This check includes the definition of the domain (horizontal and vertical extent), of the input_dates (temporal definition) and of the unit.

The corresponding transforms that may be included at this step are:

  1. regrid

  2. time_interpolation

  3. vertical_interpolation

  4. unit_conversion

For each of the above-mentioned transforms, it is possible to explicitly specify extra parameters in the related component/tracer of the datavect as follows:

datavect :
  components:
    flux:
      parameters:
        CO2:
          dir: XXX
          file: XXX
          regrid:
            method: mass-conservation

All these operations are done in the function:

pycif.plugins.obsoperators.standard.transforms.utils.init_default_transformations(self, all_transforms, backup_comps, mapper, transform, do_pipe_entry=False, trid_to_check=None)[source]

Initialize default transformations based on compatibility of input/output formats of successive transforms.

For each output trid of transform (or only trid_to_check if given) and each of its successors tr, compares the output attributes (trid_dict) against the successor’s matching input attributes (tmp_dict) and inserts, as needed, one of:

Each helper may itself splice a new transform into the pipe; this function then relies on the recursive nature of add_default.add_default() (which calls back into this function) to resolve any remaining mismatches for other dimensions, so at most one bridging transform is inserted per successor per call.

Parameters:

  • self – The parent object (obs operator), exposing datavect.

  • all_transforms – Namespace holding all registered transform instances.

  • backup_comps – Backup of components used to resolve component names that were renamed/replaced.

  • mapper – Dictionary mapping each transform id to its inputs/outputs/ precursors/successors metadata, updated when transforms are inserted.

  • transform – Id of the transform whose outputs (successors’ inputs) are being reconciled.

  • do_pipe_entry – Whether to run the pipe-entry initialization step for any newly inserted transform.

  • trid_to_check – Optional list restricting the loop to specific output trids instead of all of transform’s successors.

Debugging options#

Two options help inspect what happens at each step of the pipeline without modifying the transforms themselves:

  • save_debug — dumps the full inputs and outputs of every transform to $workdir/obsoperator/$run_id/transform_debug/. Useful for detailed inspection but slow and disk-intensive.

  • save_debug_meta — requires save_debug: True. Replaces the full NetCDF / datastore files with lightweight plain-text summaries that record dimensions, value ranges, and NaN flags. Suitable for routine debugging or large runs where a full dump would be impractical.

YAML arguments#

The following arguments are used to configure the plugin. pyCIF will return an exception at the initialization if mandatory arguments are not specified, or if any argument does not fit accepted values or type:

Optional arguments#

autorestart : bool, optional, default False

if interrupted, computations restart from the last simulated period. WARNING: the CIF cannot detect whether this period has been correctly written or is corrupt: it is necessary to check manually in the relevant directories and remove the last simulated period if a file has not been correctly written.

autoflush : bool, optional, default False

Remove big temporary files when the run is done

force-full-flush : bool, optional, default False

Complementary to autoflush. Also flushes files needed to run an adjoint. Use this option when no adjoint is needed later. The option is triggered only if autoflush is True

dump_init_state : bool, optional, default False

Dump the initial state of the observation operator in a pickle file. This is useful for debugging and for re-using the initial state in other runs. The file is saved in $workdir/obsoperator/controlvect_init.pickle and $workdir/obsoperator/obsvect/ in the case of a forward/tangent-linear run and adjoint run respectively.

save_debug : bool, optional, default False

Force transforms to save debugging information. Intermediate datastores will be saved in the directory $workdir/obsoperator/$run_id/transform_debug/

Warning

This option saves every intermediate states of the transformation pipeline. It slows drastically the computation of the obsvervation operator and can take a lot of disk space. Should be used only for debugging or understanding what happens along the way.

save_debug_meta : bool, optional, default False

Complementary to save_debug. When both save_debug and save_debug_meta are True, only lightweight plain-text metadata files are written instead of full NetCDF / datastore files. Dramatically reduces wall-time and disk overhead while still capturing enough information to trace data flow and detect anomalies.

Each text file records, per intermediate datastore entry:

  • For xr.Dataset — dimension names and sizes, min/max values, and NaN presence for the spec variable (and incr when present).

  • For pd.DataFrame — row count and, for each of the maindata, spec, and incr columns that exist, min/max values and NaN presence.

Files are written to the same $workdir/obsoperator/$run_id/transform_debug/ directory as the full debug files, but use _meta_ in their names rather than _debug_, and carry a .txt extension.

Has no effect if save_debug is False.

force_full_operator : bool, optional, default False

Force computing all transforms in the observation operator, event if no observation is to be simulated.

init_inputs : optional

Structure of components and parameters to initialize. Doing so, there is no need to define an execution mode. Only inputs that were required will be computed. Moreover, with this option, it is possible to provide a partial yaml paragraph for the datavect object: only components required to generate those required are checked before execution.

Argument structure:
any_key : optional

Name of a given component to be initialized

Argument structure:
parameters : list, optional

List of parameters to initialize for the corresponding component. Initialize all parameters if not specified

transform_pipe : optional

List of transformations to build the main observation operator pipeline

Argument structure:
any_key : optional

Name of a given transformation to be included. The name has no impact on the way the observation operator is computed, although it is recommended to use explicit names to help debugging.

Argument structure:
**args : optional

Arguments to set-up the given transform

parallel : optional

Physical parallelization of the computation of the TL and adjoint

Argument structure:
segments : str, mandatory

Length of each parallel segment

overlap : str, mandatory

Length of the initial overlap with previous segments

subprocess : bool, optional, default False

If True submit the segments in subprocesses, else submit them in new jobs with the platform plugin

nproc : int, optional

number of proc to attribute to each segments when ‘subprocess’ is True (work with LMDz only)

ref_fwd_dir : str, optional, default “”

Path to a reference forward run. This is used when using the approximate operator to accelerate its computation.

approx_operator : optional

Approximate the observation operator outside the given interval

Argument structure:
datei : str, mandatory

Start date of the interval on which to compute the real operator

datef : str, mandatory

Start date of the interval on which to compute the real operator

batch_computation : optional

Compute perturbed samples of the control vector within the same observation operator

Argument structure:
nsamples : int, mandatory

Number of samples to generate

dir_samples : str, mandatory

Directory where to fetch sample control vectors

file_samples : str, optional, default “controlvect_ensemble.pickle”

Sample control vectors file name

dont_propagate : list, optional

list of (component, parameter) tuples that should not be propagated

dont_propagate_obsvect : list, optional

list of (component, parameter) tuples that ‘toobsvect’ transformation should not be propagated

ignore_model : bool, optional, default False

Do not run the model as part of the observation operator.

force_propagate_attributes : bool, optional, default False

Force the propagation of attributes throughout transforms. Use with caution.

monitor_memory : bool, optional, default False

Print memory usage for each transform.

clean_memory : bool, optional, default True

Clean datastores that are not used anymore

autokill_time : str, optional

Stops the running simulation after a given time and re-submit it automatically in a new job. Should be one of Pandas’ offset aliases for example use ‘23h’ to stop the simulation after 23 hours. When using this option, a platform plugin with the options needed for submitting a job is required.

max_resubmissions : int, optional, default 0

Maximum number of times the simulation can be automatically re-submitted in a job.

rename_resubmit_logfile : int, optional, default True

Rename logfile for re-submitted sumulations.

onlyinit : bool, optional, default False

Does the initialization of the observation operator only

use_dask : bool, optional, default False

Use dask to manage the transform dependency graph. Requires dask to be installed (pip install dask).

dask_mode : “synchronous” or “threads” or “processes”, optional, default “synchronous”

Scheduler passed to dask.compute when use_dask is True.

Accepted values (dask canonical names and convenience aliases):

  • "synchronous" / "single-threaded" / "sync" — single-threaded, in-process execution. No parallelism, but full Python debugger support. Recommended for development and debugging.

  • "threads" / "threaded" — multi-threaded scheduler. Recommended for CIF. xarray, NumPy, SciPy and NetCDF4 all release the GIL during their core operations, so threading gives real parallelism with negligible overhead and no serialisation of Python objects. Open file handles, xarray backends and MPI state are shared safely within the same process.

  • "processes" / "multiprocessing" — multi-process scheduler.

    Warning

    Avoid this scheduler with CIF’s stack. Every task argument is pickled and sent to a worker process. This fails silently or with opaque errors for open NetCDF4 file handles, lambdas, certain xarray backends, and any object that does not support pickle. Additional hazards include fork-safety issues with inherited file descriptors and MPI communicators, and global state (e.g. logging configuration, random seeds) diverging between workers. Only consider it for transforms that are entirely self-contained and pickle-safe.

  • Any other string or object accepted by dask.compute(scheduler=…) — e.g. a dask.distributed.Client — is forwarded verbatim.

Resource knob — ``num_workers``: the only parallelism parameter exposed by the local "threads" and "processes" schedulers is num_workers (passed via dask.compute). For fine-grained per-task resource allocation (CPU sets, GPU assignment, memory limits) the dask.distributed scheduler is required.

Wrapping external executables: transforms that call Fortran or other compiled binaries should use subprocess.run (or equivalent) inside a threaded task, not switch to the processes scheduler. This keeps file-handle and MPI safety while still isolating the child process at the OS level.

  • “synchronous”: Single-threaded in-process execution (alias: ‘sync’, ‘single-threaded’)

  • “threads”: Multi-threaded scheduler — recommended for CIF (alias: ‘threaded’)

  • “processes”: Multi-process scheduler — see warning above (alias: ‘multiprocessing’)

plot_dask_graph : bool, optional, default False

Has no effect unless use_dask is True. Renders the resolved transform dependency graph as an interactive HTML page (pan/zoom/ drag, hover for task details) written to $workdir/obsoperator/$mode_$run_id/dask_graph.html. Intended for inspecting large graphs (hundreds of transforms) that a static Graphviz rendering would make unreadable.

Requires the optional networkx and pyvis packages (pip install networkx pyvis). If they are not installed, a warning is logged and the run proceeds normally without a plot.

Requirements#

The current plugin requires the present plugins to run properly:

Requirement name

Requirement type

Explicit definition

Any valid

Default name

Default version

model

Model

False

True

None

None

obsvect

ObsVect

True

True

standard

std

controlvect

ControlVect

True

True

standard

std

datavect

DataVect

True

True

standard

std

platform

Platform

True

True

None

None

YAML template#

Please find below a template for a YAML configuration:

 1obsoperator:
 2  plugin:
 3    name: standard
 4    version: std
 5    type: obsoperator
 6
 7  # Optional arguments
 8  autorestart: XXXXX  # bool
 9  autoflush: XXXXX  # bool
10  force-full-flush: XXXXX  # bool
11  dump_init_state: XXXXX  # bool
12  save_debug: XXXXX  # bool
13  save_debug_meta: XXXXX  # bool
14  force_full_operator: XXXXX  # bool
15  init_inputs:
16    any_key:
17      parameters: XXXXX  # list
18  transform_pipe:
19    any_key:
20      **args: XXXXX  # any
21  parallel:
22    segments: XXXXX  # str
23    overlap: XXXXX  # str
24    subprocess: XXXXX  # bool
25    nproc: XXXXX  # int
26  ref_fwd_dir: XXXXX  # str
27  approx_operator:
28    datei: XXXXX  # str
29    datef: XXXXX  # str
30  batch_computation:
31    nsamples: XXXXX  # int
32    dir_samples: XXXXX  # str
33    file_samples: XXXXX  # str
34    dont_propagate: XXXXX  # list
35    dont_propagate_obsvect: XXXXX  # list
36  ignore_model: XXXXX  # bool
37  force_propagate_attributes: XXXXX  # bool
38  monitor_memory: XXXXX  # bool
39  clean_memory: XXXXX  # bool
40  autokill_time: XXXXX  # str
41  max_resubmissions: XXXXX  # int
42  rename_resubmit_logfile: XXXXX  # int
43  onlyinit: XXXXX  # bool
44  use_dask: XXXXX  # bool
45  dask_mode: XXXXX  # synchronous|threads|processes
46  plot_dask_graph: XXXXX  # bool

See also

API reference (functions & code)

Additional documentation#

When use_dask is enabled, the pipeline is executed as a Dask task graph instead of pyCIF’s own sequential walk of period_order_fwd / period_order_adj. This page explains how that graph is built and run, to complement the use_dask / dask_mode / plot_dask_graph argument descriptions below.

Dask execution model#

Building the task graph#

Unlike the sequential mode, which pre-computes a single flat execution order (see fwd_adj_pipe()), the Dask mode only needs the raw precursor/successor links between (sub-simulation, transform, direction) nodes: fwd_adj_pipe stops right after computing them, storing them as self.pipe_links_fwd_dask / self.pipe_links_adj_dask.

pycif.plugins.obsoperators.standard.transforms.utils.dask.init_dask.init_dask(self, pipe_links, mode='fwd', do_simu=True, onlyinit=False, check_transforms=False, adj_test_threshold=10, save_debug=False, ignore_exceptions=False, ref_fwd_dir='', run_id=0)[source]

Build and execute the Dask task graph for one direction of the pipe.

Inverts pipe_links (which maps each node to its successors) into a dependencies mapping (node -> precursors), topologically sorts it, augments it with extra serialization dependencies coming from force_dump/force_loadout I/O constraints (update_dependencies_with_io()), then walks the sorted nodes to build one Dask delayed task per transform (add_transform()), skipping dead branches and nodes with no resolved dependency. Also attaches a “dry run” metadata dependency on the opposite-direction counterpart of each node so that side-effect-only initialization runs before the real computation.

If self.plot_dask_graph is set, writes an interactive HTML graph of the resulting task DAG (plot_task_graph()).

Finally triggers the computation of the graph’s final task (final_toobsvect in forward/tangent-linear mode, final_fromcontrol in adjoint mode), using the scheduler resolved from self.dask_mode (_resolve_dask_scheduler), restoring the root logger’s level afterwards in case a worker left it altered.

Parameters:

  • self – The obs operator, exposing workdir, transform_pipe, datei, controlvect, obsvect, plot_dask_graph and dask_mode.

  • pipe_links – Mapping of each (ddi, transform, direction) node to the list of its successor nodes, as built by fwd_pipe.fwd_adj_pipe().

  • mode – Execution mode: "fwd", "tl" or "adj".

  • do_simu – Whether to actually run the underlying model simulations.

  • onlyinit – Whether to only initialize (not execute) every transform.

  • check_transforms – Whether to keep debug information needed by the adjoint/tangent-linear consistency test.

  • adj_test_threshold – Threshold passed through to transforms for the adjoint/tangent-linear consistency test.

  • save_debug – Whether to dump debug datasets for each transform.

  • ignore_exceptions – Whether to swallow exceptions raised by transforms unless required outputs are missing.

  • ref_fwd_dir – Path to the reference forward run directory, used by adjoint/tangent-linear transforms that need forward outputs.

  • run_id – Numeric id used to name this run’s sub-directory under {self.workdir}/obsoperator/{mode}_{run_id}.

Raises:

CifImportError – If Dask is not installed.

Returns:

None. The task graph is executed for its side effects (writing to the shared datastore via each transform’s forward/ adjoint method).

init_dask inverts those links into a dependency graph, adds extra serialization edges for transforms that share files on disk (see update_dependencies_with_io()), and builds one dask.delayed() task per node (add_transform()), which in turn wraps do_individual_transform() — the actual unit of work that fetches a transform’s inputs, aggregates them, applies forward/adjoint, and re-spreads the results back to its neighbours.

Choosing a scheduler#

See the dask_mode argument below for the trade-offs between the synchronous, threads and processes schedulers. In short, threads is recommended for pyCIF’s stack (NumPy, xarray and NetCDF4 all release the GIL during their core operations), while processes should be avoided unless every transform involved is fully pickle-safe.

Visualizing the graph#

Set plot_dask_graph: True to write an interactive HTML view of the resolved graph (pan/zoom/drag, hover for task details) to $workdir/obsoperator/$mode_$run_id/dask_graph.html, via plot_task_graph(). This is intended for large graphs (hundreds of transforms) that a static Graphviz rendering would make unreadable, and requires the optional networkx and pyvis packages (pip install networkx pyvis).

Splitting the temporal grid (split_freq)#

split_freq is the framework’s main memory-scaling knob: it controls how finely the temporal axis is chopped into sub-simulations, which in turn controls how often clean_memory() gets a chance to release a piece of data. It is not a single setting — it appears independently at two unrelated places in a pipeline:

  • on a tracer/parameter, inherited from the generic datastream plugin type (split_freq argument there) — controls how input reading is chopped, upstream of fromcontrol;

  • on the toobsvect transform itself — controls how the observation vector accumulation is chopped, at the very end of the pipeline.

Both feed the same underlying mechanism (temporal splitting of input_dates/subsimus), but they act on opposite ends of the pipe and do not need to agree with each other — see below for what happens when they don’t.

The three fixed points#

Along any forward branch, three transforms declare mapper["fixed_subsimus"] = True: their sub-simulation boundaries are authoritative rather than derived from a neighbour.

Transform

What fixes its input_dates / subsimus

fromcontrol

the tracer’s own split_freq (a datastream argument), applied once by split_dates() when the data vector is initialised (init_components())

run_model

the model’s own periodicity: model.periods → model.subsimu_dates (e.g. ini_periods for ini_periods() / ini_periods())

toobsvect

its own split_freq argument, applied directly to the observation metadata via crop_monitor

Every other transform (regrid, time_interpolation, unit_conversion, vertical_interpolation, …) gets its subsimus derived, not fixed: default_subsimus() reads them off the shape of that transform’s own output input_dates — which itself got there by backward-propagation from whatever the successor needs (propagate_attributes() treats input_dates/input_files as ordinary propagated attributes). So a bridging transform’s execution granularity always follows its downstream (successor-side) need — it exists to produce that need. Its input side, meanwhile, still reads from whatever bucket the upstream fixed point gave it. This asymmetry is the whole story for memory.

Bridging mismatched grids#

When neighbouring input_dates don’t line up (different frequency, or the same frequency but a different phase), init_default_transformations() detects it and init_reindex() silently inserts a time_interpolation transform to reconcile them. This is invisible in the YAML configuration — it only shows up in transform_pipe_forward.txt / transform_pipe_adjoint.txt (dumped by dump_transform_description) or in the Dask graph plot above.

clean_memory() then, after each (ddi, transform, direction) node runs, frees a datastore slot only once no scheduled node downstream of it in ``period_order`` still needs it, as determined from pipe_links_fwd/pipe_links_adj. A bucket that is needed by many sub-simulation nodes — because it is coarser than them — stays pinned until the last of those nodes has run.

Example: LMDZ, a monthly model#

model.periods = "1MS" → run_model has one node per month (12 a year), each internally stepping at a finer resolution (e.g. daily flux input, resolved through flx_tresol). That finer resolution describes the array handed to a single monthly run_model call — it is not a finer set of pipe nodes. The one true, non-negotiable execution/memory unit on the input side is the month.

So when a tracer’s split_freq-driven bucketing does not match that monthly grid — in either direction — a time_interpolation bridge is inserted, and (per the rule above) that bridge’s own subsimus are forced onto the successor’s grid, i.e. monthly, since that is what run_model needs. That is what actually governs when clean_memory can release the data — not the tracer’s own split_freq:

Input split_freq

vs. model.periods (monthly)

Effect

"1MS" (monthly)

matches exactly

direct fromcontrol → run_model link, no bridge inserted; released right after the one LMDZ month that consumes it

"1D" (daily)

finer, but still nests inside one month

a time_interpolation bridge is inserted, forced onto the monthly grid on its output side; its input side gathers that month’s ~30 daily fromcontrol buckets before handing off. Released once per LMDZ month too — the same cadence as monthly split_freq, just with one extra transform in the pipe. Finer splitting buys smaller/more numerous reads on the fromcontrol side, but no extra memory benefit, since retention is still capped by the one month that consumes it

"1YS" (yearly)

coarser than the monthly unit

same bridge, but now the single yearly bucket is the only precursor for all 12 monthly run_model nodes: held in memory across the entire year, released only once the 12th (last) LMDZ month referencing it has run

Symmetrically downstream: a toobsvect.split_freq coarser than LMDZ’s monthly output (e.g. yearly) forces the sampling bridge feeding toobsvect to wait for all 12 monthly LMDZ outputs before it can assemble that yearly observation bucket — memory held until the 12th month is done, mirroring the input-side case exactly.

Example: CHIMERE, a daily model#

model.periods = "1D" → run_model has one node per day (365 a year), the finest fixed point in a typical pyCIF pipeline, each internally sub-stepping hourly.

Input split_freq

vs. the daily unit

Effect

"1D" (daily)

matches

direct link; one day’s emission/meteo file loaded, used by exactly one run_model day, freed right after — peak memory ≈ one day of that stream

"1MS" (monthly)

coarser

a time_interpolation bridge runs daily on the output side to satisfy CHIMERE, but the monthly input bucket (e.g. a monthly-average boundary condition) has to stay resident across all ~30 daily run_model nodes of that month before release

And on the observation side (toobsvect.split_freq, independently):

toobsvect.split_freq

Effect

"1D" (daily)

closely tracks CHIMERE’s own daily cadence; the sampling bridge feeding toobsvect is itself daily, so obsvect.ysim/obsvect.dy get updated and the corresponding simulated slice released one day at a time — the best-case scaling, peak memory ≈ one day of model output plus one day of observations

"1MS" (monthly)

toobsvect’s own subsimus are fixed at monthly — it is one of the three authoritative points, it cannot be forced finer. The bridge feeding it must accumulate CHIMERE’s daily-resolution point-sampled output across the whole month before it can hand a complete monthly chunk to toobsvect, so a month’s worth of sampled concentrations has to stay in memory even though CHIMERE itself produced it one day at a time

Because CHIMERE’s own fixed point already sits at daily resolution, matching split_freq to it on either flank genuinely buys per-day release — there is no hidden coarser fixed point absorbing the benefit, unlike LMDZ’s monthly cadence above.

Note

Whether a mismatch is bridged by blending proportionally across overlapping sub-periods or by picking the sub-period with the largest overlap is controlled by time_interpolation’s own recombine_periods argument (default True). CHIMERE, for instance, explicitly sets it to False for its surface fields, since blending emission rates across day boundaries would not be physically meaningful there.

Rule of thumb#

The memory-release cadence for a given data stream is governed by the periodicity of the nearest fixed point on the consuming/producing side (run_model’s model.periods on the input flank, toobsvect itself on the observation flank) — not by that stream’s own split_freq:

  • split_freq equal to the fixed point’s own period → optimal, no bridge inserted.

  • split_freq finer than the fixed point’s period → release-neutral (still capped by the fixed point), costs one extra bridging transform.

  • split_freq coarser than the fixed point’s period → actively worse: retention stretches to span however many of the fixed point’s sub-simulations the coarse bucket covers.