##############################
How to make super-observations
##############################

Tutorial on how to make super-observations from a set of data and for a given set-up of a CTM.

Super-observations are data to be assimilated which are built from a set of more numerous data. Super-observations can be averages, medians or other combinations of these "raw" data.

Making super-observations is a way to reduce the number of data to assimilate while keeping a relevant amount of information and, potentially, avoiding the assimilation of incompatible information or of redundant information.

The computation of super-observations is not included as such in the CIF as it requires to be carefully thought out for each case.

Nevertheless, following the main steps described here ensures an optimal use of the CIF's capacities and a good consistency in the various studies.

1. Run a forward simulation with the so-called raw data.
--------------------------------------------------------

XXX renvoyer a comment faire + points importants = bien ajuster la config du modele = a la fin de cette simu, tout doit etre fixe sauf le jeu d'obs XXX

2. Use the output monitor to select data
-----------------------------------------

From this first simulation, a monitor file is generated: :bash:`$WORKIDR/obsoperator/fwd_0000/obsvect/satellites/SPEC/monitor.nc`. It contains information computed by the CIF with the configuration of the CTM: the spatial coordinates of the grid cell matching each data (i, j, level), its temporal coodinates (tstep, tstep\_glo, dtstep) and the equivalent of the observation (sim). 

In case of satellite data, an information file is also generated to keep track of the specific data associated to each :bash:`obs`: :bash:`$WORKDIR/obsoperator/fwd_0000/chain/satellites/default_00001/infos_DATEDEB.nc` with DATEDEB the date of the beginning of the simulation XXwhy??XX.

This complete set of information makes it possible to filter out data according to criteria taking into account the simulation e.g. filtering out data with too large a discrepancy with the simulation. 

XXX example codes for plotting the data+sim?XXXX

An example code to use monitor and info together, apply some filtering and changes and generate a "clean" monitor is provided here:

     .. container:: toggle

       .. container:: header

           Show/Hide Code

       .. literalinclude:: /python_examples/filter_monitor_TROPOMI.py
          :language: python


XXX what about le bouchage de trous??XXXX

From this clean monitor, it may be useful to re-run a forward simulation, same as in step 1.

3. Generate super-observations
------------------------------

From an output monitor, it is easy to combine data at given spatial and temporal scales,
either in the data's world (using lat, lon, date) or in the model's grid (using i, j, tstep\_glo).
Using the matching info file, a super-obs input monitor can then be generated.
Using the same type of code as above, the super-observations can be generated from grouping
on the dataframe containing the full information i.e. :bash:`ds3` (after or instead of lines 40-44)
then performing the retrieving of the relevant satellite specific information in the same way.

XXX put example codes XXX

4. Use the super-observations
-----------------------------

Use the monitor containing the super-observations for a forward simulation and a adjoint test.

If everything is normal, it can then be used as input for inversions **with the same configuration of the CTM**.