General principles of using transformations: examples for building the inputs of a CTM and/or elaborating on its outputs

General principles of using transformations: examples for building the inputs of a CTM and/or elaborating on its outputs#

Main principles#

The datavect components do not need to map directly to the inputs or outputs of the CTM driven by the CIF.

Various elementary transformations are available and can be combined to go from the components to the actual CTM inputs, or to combine CTM outputs into observation equivalents. Custom (chains of) transformations are defined via controlvect for processing CTM inputs (more generally, transformations at the beginning of the CIF’s assimilation chain) and via obsvect for processing CTM outputs (transformations at the end of the chain). Transformations are specified using the transform_pipe key.

Main steps for defining personalized transformations#

For building the inputs of the CTM#

We take here the example of fluxes for a forward simulation with CHIMERE.

In datavect, define the component containing the original files, with the usual information on how to interpolate them onto the CTM grid.

In the following example, two raw emission sources are used for two methane-emitting sectors.

datavect:
  plugin:
    name: standard
    version: std

  components:
    meteo:
      plugin:
        name: CHIMERE
        version: std
        type: meteo
      dir: /home/chimereicos/espigrad_reanalysis/inputs/meteo_from_old/
      file: METEO.%Y%m%d%H.24.nc
      file_freq: 1D

    flux:
      parameter:
        CO2:
          plugin:
            name: CHIMERE
            version: AEMISSIONS
            type: flux
          dir: /home/chimereicos/espigrad_reanalysis/inputs/emis_from_independent/
          file: AEMISSIONS.2019%m28%H.24.nc
          file_freq: 1D

    sectorflux:
      parameters:
        CH4emisS10:
          plugin:
            name: CHIMERE
            version: AEMISSIONS
            type: flux
          dir: /home/chimereicos/espigrad_reanalysis/inputs/emis_from_independent/
          file: LNAEMISSIONS.2019%m28%H.24.nc
          file_freq: 1D
          varname: LOGSNAP10
        CH4emisS9:
          plugin:
            name: EDGAR
            version: v5
            type: flux
          dir: /tmp/PYCIF_DATA_TEST/RAW/EMISSIONS/EDGARV5/TOTAL
          regrid:
            method: mass-conservation
          time_interpolation:
            method: linear
          unit_conversion:
            scale: 1.368e+21
          vertical_interpolation:
            method: closest
            fill_nans: false
            fill_nans_value: 0
          varname: emi_ch4_snap9
          file: 'v50_N2O_%Y.0.1x0.1.nc'

Note that:

in the example, CO2 uses ready-made files, while CH4 must be built from the sectorflux component.
component names can be freely chosen by the user
parameter names can be freely chosen by the user

In controlvect, add the transform_pipe that defines the transformations to apply.

controlvect:
  plugin:
    name: standard
    version: std
    type: controlvect
  save_out_netcdf: True
  transform_pipe:
      TBC

Use the elementary transformations to specify the chain of custom transformations that convert datavect components into the actual CTM inputs.

controlvect:
  plugin:
    name: standard
    version: std
    type: controlvect
  save_out_netcdf: True
  transform_pipe:
    take_exp: # take the exponential of emissions provided as ln
      plugin:
        name: exp
        type: transform
        version: std
      component: sectorflux
      parameters_in:
      - CH4emisS10
    take_sum_emis: # sum various sectors to obtain the total
      plugin:
        name: families
        version: std
        type: transform
      component: sectorflux
    parameters_in: [ CH4emisS10, CH4emisS9]
    component_out: flux
    parameter_out: CH4

Note

The CIF applies transformations in the order they appear in the YAML file. In the example, CH4emisS10 is exponentiated first, then the result is summed with CH4emisS9.

Note that:

transformation names are freely chosen by the user (use descriptive names as a best practice)
the arguments for each transformation type are documented here
component specifies the input component for the transformation, and must match a name defined in datavect
after applying the transform_pipe, the resulting component names must match the required CTM input names; therefore, the final component_out in a chain must match the component name expected by the CTM

Run XX with only_init?XX and check with the files for checks that the personalized transforms in controlvect do what is expected.
Run the full forward.XX wchich checks?XX

For combining the outputs of the CTM#

We take here the example of isotopic data for a forward simulation.

In datavect, define the component containing the observation files to compare with the simulation.

In the following example, isotopic methane data are used. They are already formatted as a CIF input monitor file.

datavect:
  plugin:
    name: standard
    version: std
  components:
  concs:
    dir: ${PYCIF_DATATEST}/LMDZ//clumped_isotopes//monitor/
    file: monitor_filtered_rsd_new.nc
    file_statlev: ${PYCIF_DATATEST}/LMDZ//simple/stations_levels_LMDZ39lev.txt
    parameters:
      CH4: {}
      d13C: {}

Note that :XXXX what are the parameters???XXX

In obsvect, add the transform_pipe that defines the transformations to apply.

obsvect:
  plugin:
    name: standard
    version: std
  dump: True
  transform_pipe:
      TBC

Use the elementary transformations to specify the chain of custom transformations that produce observation-equivalent values from CTM outputs.

obsvect:
  plugin:
    name: standard
    version: std
  dump: True
  transform_pipe:
      makeiso:
          plugin:
              name: conc2ratio
              type: transform
              version: std
       component:
       - concs
       - concs
            XXXXXXXXXXXXXXXXXXXXXXXXX
         parameters_in:
      names:
      - 12CH4
      - 13CH4
    parameters_out:
      d13C:
        isotopologues:
        - 13CH4
        refs:
        - 12CH4
        standard: 0.0112372
      iso_mass:
      - 16.031
      - 17.035
      names:
      - d13C
      - CH4
      spec_mass: 16.0425

    unit: volume