pycif.plugins.obsvects.standard — API reference

pycif.plugins.obsvects.standard — API reference#

Configuration reference: standard plugin

pycif.plugins.obsvects.standard.build_full_r.build_r(obsvect, **kwargs)[source]#

Build the full observation error covariance matrix \(\mathbf{R}\).

Constructs the square matrix \(\mathbf{R} \in \mathbb{R}^{m \times m}\) by filling diagonal blocks for each observation tracer with the squared per-observation errors:

\[R_{ii} = \sigma_{\varepsilon,i}^2 \quad \forall\, i \in [\text{ypointer},\, \text{ypointer} + \text{dim})\]

Off-diagonal elements remain zero (diagonal covariance assumption).

Warning

This returns a dense \(m \times m\) matrix. For large observation vectors (e.g. \(m > 10^4\)) this will be memory-prohibitive. Use rinvprod() for the efficient diagonal application.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance (provides yobs_err, dim, and access to the datavect tracer metadata).

  • **kwargs – unused; accepted for interface consistency.

Returns:

diagonal matrix of shape (dim, dim) containing the squared observation errors on the main diagonal.

Return type:

np.ndarray

pycif.plugins.obsvects.standard.fetch.default_fetch(ref_dir, ref_file, input_dates, target_dir, tracer=None, **kwargs)[source]#

Resolve observation monitor files and symlink them to the run directory.

For each sub-simulation date in input_dates, expands ref_dir and ref_file using strftime formatting, creates a symlink from the source file to target_dir, and returns de-duplicated sorted lists of local file paths and their associated dates.

Parameters:

  • ref_dir (str) – directory template for the source files; may contain strftime format codes (e.g. /data/%Y/%m).

  • ref_file (str) – file name template; may contain strftime codes (e.g. monitor_%Y%m%d.nc).

  • input_dates (dict) – mapping from sub-simulation start dates to lists of dates for which files should be fetched.

  • target_dir (str) – local run directory into which symbolic links are written; the base name of each source file is preserved.

  • tracer – unused; accepted for interface consistency with other fetch functions.

  • **kwargs – unused; accepted for interface consistency.

Returns:

a pair (list_files, list_dates) where both are dicts keyed by the same sub-simulation start dates as input_dates. Each value is a sorted, de-duplicated list of local file paths or resolved datetime objects respectively.

Return type:

tuple[dict, dict]

pycif.plugins.obsvects.standard.ini_mapper.ini_mapper(obsvect, general_mapper={}, backup_comps={}, transforms_order=[], ref_transform='', **kwargs)[source]#

Build the transform mapper for the observation vector.

Scans the datavect components and collects every (component, tracer) pair whose isobs flag is True. These pairs become the output tracer IDs of the observation operator — i.e. the quantities that the toobsvect system transform will write into obsvect.ysim.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance (carries the populated datavect with component/tracer metadata).

  • general_mapper (dict) – mapper dictionaries from other transforms; unused but accepted for interface consistency.

  • backup_comps (dict) – unused; accepted for interface consistency.

  • transforms_order (list) – unused; accepted for interface consistency.

  • ref_transform (str) – unused; accepted for interface consistency.

  • **kwargs – unused.

Returns:

mapper dict with "inputs": {} (observation vector has no inputs from other transforms) and "outputs": {(comp, trcr): …} for every observation tracer.

Return type:

dict

pycif.plugins.obsvects.standard.init_rinvprod.init_rinvprod(obsvect, measurements, **kwargs)[source]#

Sanitise observation errors before the \(\mathbf{R}^{-1}\) product.

Replaces any non-positive (zero or negative) obserror values in the datastore with the dataset mean, preventing rinvprod() from encountering divisions by zero.

Note

Transport error inflation is mentioned in comments as a future extension; at present only the zero-error replacement is implemented.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance (carries datastore with an obserror column).

  • measurements (Plugin) – unused; kept for API compatibility with the pyCIF plugin interface.

  • **kwargs – unused; accepted for interface consistency.

pycif.plugins.obsvects.standard.init_y0.init_y0(obsvect, **kwargs)[source]#

Initialise the flat observation-vector arrays from the datavect configuration.

Iterates over every component / tracer pair declared in obsvect.datavect.components and, for each one that is flagged as an observation (tracer.isobs):

  1. Loads the datastore from a pre-computed monitor.nc file (dir_obsvect is set) or reads it from the raw monitor files via init_param().

  2. Assigns a contiguous slice in the global yobs / yobs_err / ysim / dy arrays using tracer.ypointer.

  3. Appends the obsvect_mask boolean array that marks which observations enter the cost function (is_obsvect == True).

  4. Optionally applies per-tracer error scaling (obserror_scale and obserror_value).

  5. Compresses the tracer datastore to save memory.

  6. Dumps the final observation vector when obsvect.dump_obs is set.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance. On entry its flat arrays (yobs, ysim, etc.) are empty; on exit they are filled.

  • **kwargs – forwarded to init_param() and downstream datastream read / fetch methods.

Returns:

the updated obsvect instance (also modified in-place).

Return type:

Plugin

pycif.plugins.obsvects.standard.rinvprod.rinvprod(obsvect, dy: ndarray[tuple[Any, ...], dtype[floating]], inverse: bool = True, mask: ndarray[tuple[Any, ...], dtype[bool]] | None = None) → ndarray[tuple[Any, ...], dtype[floating]][source]#

Apply the observation error covariance (or its inverse) to a vector.

Assumes a diagonal observation error covariance matrix \(\mathbf{R} = \mathrm{diag}(\sigma_{\varepsilon,1}^2, \ldots, \sigma_{\varepsilon,m}^2)\).

Two modes of operation:

  • inverse=True (default) — computes \(\mathbf{R}^{-1}\,\delta\mathbf{y}\):

    \[(\mathbf{R}^{-1}\,\delta\mathbf{y})_i = \frac{\delta y_i}{\sigma_{\varepsilon,i}^2}\]

    Used in the cost function gradient: \(\nabla J_o = \mathbf{H}^\top \mathbf{R}^{-1} (\mathcal{H}(\mathbf{x}) - \mathbf{y}^o)\).

  • inverse=False — computes a noise-perturbed observation sample \(\mathbf{y}^o + \mathbf{R}^{1/2}\,\delta\mathbf{y}\):

    \[(\mathbf{R}^{1/2}\,\delta\mathbf{y} + \mathbf{y}^o)_i = \sigma_{\varepsilon,i}\,\delta y_i + y^o_i\]

    Used for Monte Carlo sampling of perturbed observations.

An optional mask restricts the operation to the active observation subset (obsvect.obsvect_mask); masked-out positions are set to zero in the output.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance (provides yobs_err and yobs).

  • dy (np.ndarray) – input vector, shape (dim,).

  • inverse (bool) – if True apply \(\mathbf{R}^{-1}\); if False apply \(\mathbf{R}^{1/2}\) and add yobs.

  • mask (np.ndarray of bool, optional) – boolean mask of shape (dim,) selecting a subset of observations. Positions where the mask is False are set to zero in the output.

Returns:

result vector, shape (dim,).

Return type:

np.ndarray

Raises:

ValueError – if mask has a different shape than dy, or if any entry of yobs_err is zero or NaN (which would make \(\mathbf{R}^{-1}\) undefined or infinite).

pycif.plugins.obsvects.standard.utils.check_monitor.check_monitor(self, tracer)[source]#

Check whether a pre-existing monitor file is compatible with the current run.

Compares the domain dimensions and simulation dates stored in the monitor’s nc_attributes (or .attrs) against the current model configuration. This allows init_param() to decide whether to re-use the monitor as-is or to reload and re-crop the raw observations.

Parameters:

  • self (Plugin) – obsvect plugin instance (provides model.domain, datei, and datef).

  • tracer (Plugin) – datavect tracer plugin instance whose datastore contains the pre-existing monitor data.

Returns:

a four-element tuple (allcorrect, ok_hcoord, ok_vcoord, do_tstep) where:

  • allcorrect — all checks passed; the monitor can be used as-is.

  • ok_hcoord — horizontal domain (nlat, nlon) matches.

  • ok_vcoord — vertical coordinate is consistent (always True in the current implementation — reserved for future use).

  • do_tstep — temporal re-cropping is required (True when dates do not match).

When the monitor has no nc_attributes (old format), returns (False, False, False, True) to trigger a full reload.

Return type:

tuple[bool, bool, bool, bool]

pycif.plugins.obsvects.standard.utils.dump.dump(obsvect, dir_dump)[source]#

Write the populated observation vector to monitor.nc files.

For each observation tracer, creates a copy of its datastore, fills the maindata columns (obs, obserror, sim, sim_tl) from the flat obsvect arrays, optionally adds an orig_index column, and serialises the result to:

<dir_dump>/<component>/<tracer>/monitor.nc

The file format is controlled by obsvect.dump_type ('nc' or 'csv'). Global NetCDF attributes recording the run dates and model configuration are written as nc_attributes.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance. Must have been initialised (yobs, ysim, dy, yobs_err filled).

  • dir_dump (str) – root directory under which the per-tracer sub- directories are created.

pycif.plugins.obsvects.standard.utils.read.read(obsvect, dir_dump)[source]#

Reload simulation results from previously dumped monitor.nc files.

Reads the sim, sim_tl, obs, and obserror columns from the per-tracer monitor.nc files written by dump() and copies them back into the flat obsvect arrays (ysim, dy, yobs, yobs_err). Also restores tracer.datastore for each tracer.

This is used to reload forward-run results without re-running the observation operator (e.g. for adjoint restarts or post-processing).

Parameters:

  • obsvect (Plugin) – obsvect plugin instance with dim, yobs, ysim, dy, yobs_err already allocated (typically after init_y0()).

  • dir_dump (str) – root directory from which per-tracer monitor files are read (<dir_dump>/<component>/<tracer>/monitor.nc).

Returns:

True if no observation tracers were found (nothing to read); False if at least one tracer was successfully reloaded.

Return type:

bool

pycif.plugins.obsvects.standard.utils.vcoord.vcoord(obsvect, **kwargs)[source]#

Assign a model vertical level to each observation.

Dispatches to one of two strategies based on whether a pre-defined station level file is available:

  • vcoordfromfile() — when obsvect.file_statlev is set, reads a text table mapping station names to fixed level indices.

  • vcoordfrommeteo() — otherwise, derives the level by matching each observation’s altitude against a hard-coded mid-layer altitude array (LMDZ-specific, provisional implementation).

The level column of obsvect.datastore["data"] is filled in-place.

Parameters:

  • obsvect (Plugin) – obsvect plugin instance (carries datastore, optionally file_statlev, and workdir).

  • **kwargs – forwarded to the chosen sub-function.

Returns:

the updated obsvect (also modified in-place).

Return type:

Plugin

pycif.plugins.obsvects.standard.utils.vcoord.vcoordfromfile(datastore, file_lev, **kwargs)[source]#

Assign model levels to observations using a pre-defined station table.

Reads a whitespace-delimited file (header line skipped) whose columns are:

STAT   LAT    LON   LEV   alt(m)

and sets datastore["level"] to the model level index (column 4, 0-based Python indexing) for each matching station name. Observations from stations not listed in the file retain NaN.

Parameters:

  • datastore (pd.DataFrame) – observation dataframe with a "station" column (station names, lower-case).

  • file_lev (str) – path to the station-level file.

  • **kwargs – unused; accepted for interface consistency.

Returns:

datastore with the "level" column updated in-place.

Return type:

pd.DataFrame

pycif.plugins.obsvects.standard.utils.vcoord.vcoordfrommeteo(workdir, datastore, **kwargs)[source]#

Assign model levels to observations by matching altitudes to LMDZ mid-layers.

Uses a hard-coded array of LMDZ mid-layer altitudes (39 levels, metres above the surface) and assigns each observation to the nearest level by minimising \(|\text{alt}_{obs} - \text{alt}_{level}|\).

Warning

This function is provisional and LMDZ-specific. The altitude array is hard-coded and the meteo directory ($workdir/meteo/) is not actually used in the current implementation. A commented-out block shows the intended xarray-based generalisation.

Parameters:

  • workdir (str) – pyCIF working directory (currently unused; reserved for the future generalised implementation).

  • datastore (pd.DataFrame) – observation dataframe with an "alt" column (altitude in metres above the surface).

  • **kwargs – unused.

Returns:

datastore with the "level" column filled with 1-based LMDZ level indices (closest level wins).

Return type:

pd.DataFrame