RFC-0010: Decouple control/observation vector operations from datavect via a dedicated operator transform

RFC-0010: Decouple control/observation vector operations from datavect via a dedicated operator transform#

Field

Value

Author(s)

Antoine Berchet

Status

Draft

Created

2026-07-29

Decided

Reviewers

Supersedes

— (follow-up to ADR-0009, does not replace it)

Superseded by

—

Summary#

CIF’s control vector is necessarily gridded today, while the observation side already supports sparse data — an asymmetry that blocks inherently sparse/point-like control vectors (land-process-model parameters, point sources). Both vectors are wired to datavect (per ADR-0009), which only recognises data the user explicitly declares in YAML: an observation operator’s internal intermediate data has no way to be configured, and observation vector entries built from several data points (“super-observations” — horizontal lidars, SVD/encoder-compressed representations) can’t be represented at all. This RFC proposes splitting today’s transform pipeline into three explicit layers — a datavect I/O transform, a merged control/observation operator transform with explicit forward and adjoint, and a thin fromcontrol/toobsvect interface to the raw controlvect.x / obsvect.ysim arrays — so that sparse control vectors and explicitly-configurable intermediate transforms become representable without depending on datavect for everything. The concrete super-observation data structure is intentionally left to a follow-up RFC; this RFC scopes the enabling architecture only, and is designed to be additive/opt-in so existing configs are unaffected.

Motivation / Problem#

  • Control vector is necessarily gridded, observation side is already sparse. This asymmetry blocks control-vector use cases that are inherently sparse/point-like: parameter optimization for land-process models (e.g. per-site or per-PFT parameters), and point-source inversions (a handful of discrete sources, not a grid cell field).

  • Intermediate transform outputs can’t be configured when observation operators are complex. If transform 1 produces data1 which feeds transform 2, there is no way to specify operator-specific choices on data1 (e.g. which regridding method to apply) because data1 isn’t declared in datavect. Per ADR-0009, obsvect/controlvect (and the auto-wiring in init_sparse.py) only recognise what’s declared in datavect — anything not declared there has no way to be configured or referenced explicitly.

  • Super-observations aren’t representable. A single observation-vector entry built from several underlying data points doesn’t exist as a concept today. This covers both genuinely complex observation geometries (horizontal-viewing lidars) and derived / compressed representations (SVD components, learned encoder outputs). obsvect assumes a flat one-row-per-point sparse structure with no aggregation concept, which also blocks cluster-based analyses, satellites observed at multiple scales, geopandas-style complex observation geometries, and assimilating gridded data directly as observations.

  • These are exactly the two “known weaknesses” ADR-0009 flagged: no element can exist in the control/observation vector without a datavect entry, and vector membership is implicit and error-prone. This RFC is the follow-up ADR-0009 anticipated.

Context & constraints#

  • datavect centralisation (ADR-0009). controlvect/standard and obsvect/standard both declare datavect as a hard requirement and derive their structure from it — there is no independent declaration of vector content today.

  • Mapper / auto-wiring architecture (ADR-0002). Transforms exchange data through a shared mapper dict keyed by (component, parameter); init_sparse.py compares a precursor’s sparse_data/sampled/continuous_*domain flags against its successor’s and auto-inserts the needed conversion transform. This machinery currently only sees mapper entries that trace back to a datavect declaration.

  • Current fromcontrol / toobsvect responsibilities. These system transforms today conflate two things: interfacing with the raw controlvect.x / obsvect.ysim arrays, and carrying out control/observation-specific operations (resolution handling, operator-specific processing). This RFC’s core move is separating those two responsibilities.

  • Plugin interface contract (ADR-0006). Any new transform “kind” introduced here still goes through the single Plugin base class / plugin_types registry — this RFC does not touch the extensibility mechanism itself, only what happens inside the transform plugin category.

Proposed design#

Split the pipeline into three explicit layers, replacing the current entanglement of datavect membership with control/observation-specific operations:

  1. datavect I/O transform — pure fetch (control-vector direction: read the prior/background data a plugin needs) or store (observation-vector direction: write data back for bookkeeping/output). No operator logic lives here.

  2. A merged control/observation operator transform, with explicit forward and adjoint — absorbs what’s split today between controlvect-side “resolution” operations and obsoperator-specific operations into one transform kind shared by both sides. Because this layer is decoupled from datavect, its intermediate data (the data1 example above) can be explicitly configured — e.g. which regridding to use — without requiring a datavect entry for it.

  3. fromcontrol / toobsvect, narrowed to a thin interface — only the plumbing that connects the operator-transform layer to controlvect.x / obsvect.ysim.

This decoupling is what makes sparse control vectors and (eventually) super-observations representable: neither is forced through the current datavect-shaped bottleneck, because the operator transform’s internal data no longer needs a datavect entry to exist or be configured.

Explicitly out of scope for this RFC: the concrete super-observation data structure (how a multi-point aggregate, SVD component, or encoder output is represented inside obsvect, and how its forward/adjoint is defined). This RFC only removes the architectural blocker (datavect-forced 1:1 membership); the schema itself is deferred to a follow-up RFC once this layering is in place.

Additive / opt-in: existing configs that rely on today’s implicit datavect-driven wiring keep working unchanged. The new operator-transform layer is something a config opts into when it needs sparse control-vector elements, explicit intermediate-transform configuration, or (later) super-observations.

Alternatives considered#

  • Do nothing / keep working around it in ad hoc plugin code. Rejected: this is exactly the implicit-ambiguity failure mode ADR-0009 already flagged, and it gets worse as more complex observation operators are added.

  • Extend the datavect schema instead (e.g. an internal: true flag letting an entry exist without being user-facing). A lighter-weight alternative that would address the intermediate-transform problem alone, without touching fromcontrol/toobsvect or enabling sparse control vectors or super-observations. Not chosen as the primary design here because it doesn’t address the control-vector asymmetry, but flagged in Risks below as worth reviewers weighing in on — it could be a smaller complementary fix or a staging step.

  • Build super-observations as a one-off obsvect feature, without generalising the transform layer. Rejected: it wouldn’t solve the sparse-control-vector or intermediate-transform problems, and would likely need to be redone once those are addressed anyway.

Invariants & assumptions affected#

  • [x] Adjoint / tangent-linear consistency — the new merged operator transform must supply both forward and adjoint; existing gradient-check infrastructure (check_adjtltest.py) needs to cover it explicitly.

  • [x] Numerical reproducibility — existing reference configurations must reproduce bit-for-bit, since the change is additive/opt-in and doesn’t touch the code path configs don’t opt into.

  • [ ] Mass conservation / physical invariants — N/A unless the operator transform introduces new resampling; existing regrid/interpolation invariants must still hold when that logic moves into the new operator layer.

  • [x] Plugin interface contract — the transform plugin category’s interface changes (narrower fromcontrol/toobsvect, new operator-transform kind); existing transform plugins (regrid, vertical_interpolation, etc.) need to be evaluated against the new layering.

  • [x] Config / YAML backward compatibility — by design, existing configs are unaffected; new capabilities require new YAML. No migration should be needed for current users.

  • [ ] On-disk / output format — N/A for this RFC; may become relevant once super-observations are designed in the follow-up RFC.

  • [x] Ordering dependencies / global state — the new operator layer sits between the datavect I/O transform and fromcontrol/toobsvect; init_sparse.py’s current auto-wiring insertion order must either be preserved for the unchanged path or explicitly superseded for the opt-in path. Needs to be resolved during design, not assumed.

Impact#

  • Affected components / plugins / instruments: controlvects/standard, obsvects/standard, transforms/system/fromcontrol, transforms/system/toobsvect, transforms/utils/init_sparse.py (auto-wiring), datavects/standard.

  • Performance & memory: expected negligible if the new layer is implemented as another task in the existing Dask graph; to be confirmed during implementation.

  • User-facing changes: new, additive YAML syntax to declare sparse control-vector elements and to explicitly configure intermediate transforms; nothing changes for configs that don’t use the new syntax.

  • Documentation that must change: docs/decisions/0009 stays as-is (it accurately records the state before this RFC); new user/dev docs will be needed for the operator-transform layer once implemented, alongside the existing datavect/controlvect/obsvect tutorials.

Validation plan#

  • Gradient / finite-difference adjoint checks on the new merged operator transform, reusing the existing check_adjtltest.py infrastructure.

  • Regression run on existing reference configurations to confirm bit-identical results (the opt-in guarantee above).

  • A small point-source or land-process-parameter benchmark exercising the new sparse control-vector path once implemented, since that’s the first concrete use case this unblocks.

Risks & open questions#

  • Whether the “extend datavect schema” alternative should be pursued in addition to this restructuring, as a lighter complementary fix for the intermediate-transform problem specifically — open for reviewers.

  • The concrete super-observation schema is deferred to a follow-up RFC; candidate use cases to keep in mind when designing it: cluster-based analyses, satellites observed at multiple scales, geopandas-style complex observation geometries, and assimilating gridded data directly as observations.

  • How the new operator-transform layer interacts with today’s sparse_data/sampled flag-driven auto-wiring (ADR-0002) — replace, wrap, or coexist? Needs resolution during implementation design, not assumed here.

  • This touches core pipeline-construction / mapper-wiring code, one of the most central and highest-blast-radius parts of the codebase — implementation should be reviewed with that in mind.

Rollout / migration#

Phased, and additive throughout — no forced migration for existing users:

  1. Implement the datavect I/O transform and the merged operator transform layer; existing gridded/sparse configs continue through their current (unchanged) path.

  2. Add sparse control-vector support (point sources, land-process-model parameters) as the first concrete use case exercising the new layer.

  3. Revisit super-observations in a follow-up RFC once (1)–(2) are in place and validated.

Decision#

Outcome: Date / decider: Rationale (esp. if rejected or changed): Follow-up issues / MRs: