##############################
How to make super-observations
##############################

This tutorial explains how to build super-observations from a dataset for a given CTM configuration.

Super-observations are data built for assimilation by aggregating a larger set of raw measurements. They can be averages, medians, or other combinations of the raw data.

Building super-observations reduces the number of data points to assimilate while preserving relevant information and, potentially, avoiding the assimilation of incompatible or redundant measurements.

The computation of super-observations is not handled directly by the CIF, as it must be carefully designed for each case.

However, following the main steps described here ensures optimal use of the CIF's capabilities and consistency across studies.

1. Run a forward simulation with the so-called raw data.
--------------------------------------------------------

XXX renvoyer a comment faire + points importants = bien ajuster la config du modele = a la fin de cette simu, tout doit etre fixe sauf le jeu d'obs XXX

2. Use the output monitor to select data
-----------------------------------------

This first simulation generates a monitor file at :bash:`$WORKDIR/obsoperator/fwd_0000/obsvect/satellites/SPEC/monitor.nc`. It contains information computed by the CIF using the CTM configuration: the spatial coordinates of the grid cell matching each data point (i, j, level), its temporal coordinates (tstep, tstep\_glo, dtstep), and the simulated equivalent of the observation (sim).

For satellite data, an additional information file is generated to record the specific data associated with each :bash:`obs`: :bash:`$WORKDIR/obsoperator/fwd_0000/chain/satellites/default_00001/infos_DATEDEB.nc`, where DATEDEB is the start date of the simulation.

This complete set of information allows filtering data based on simulation criteria, for example removing data points with too large a discrepancy from the simulation.

XXX example codes for plotting the data+sim?XXXX

An example code to use monitor and info together, apply some filtering and changes and generate a "clean" monitor is provided here:

     .. container:: toggle

       .. literalinclude:: /python_examples/filter_monitor_TROPOMI.py
          :language: python


XXX what about le bouchage de trous??XXXX

From this clean monitor, it may be useful to re-run a forward simulation, same as in step 1.

3. Generate super-observations
------------------------------

From an output monitor, it is straightforward to combine data at given spatial and temporal scales,
either in observation space (using lat, lon, date) or in model grid space (using i, j, tstep\_glo).
Using the matching info file, a super-observation input monitor can then be generated.
Using code similar to the example above, super-observations can be generated by grouping
the full-information dataframe (i.e. :bash:`ds3`, after or instead of lines 40–44)
and retrieving the relevant satellite-specific information in the same way.

XXX put example codes XXX

4. Use the super-observations
-----------------------------

Use the monitor containing the super-observations for a forward simulation and an adjoint test.

If everything looks correct, it can then be used as input for inversions **with the same CTM configuration**.