########################################################################################################################## General principles of using transformations: examples for building the inputs of a CTM and/or elaborating on its outputs ########################################################################################################################## Main principles ---------------- The :bash:`datavect` components do not need to map directly to the inputs or outputs of the CTM driven by the CIF. Various :doc:`elementary transformations` are available and can be combined to go from the components to the actual CTM inputs, or to combine CTM outputs into observation equivalents. Custom (chains of) transformations are defined via :bash:`controlvect` for processing CTM inputs (more generally, transformations at the beginning of the CIF's assimilation chain) and via :bash:`obsvect` for processing CTM outputs (transformations at the end of the chain). Transformations are specified using the :bash:`transform_pipe` key. Main steps for defining personalized transformations ----------------------------------------------------- For building the inputs of the CTM ................................... We take here the example of fluxes for a forward simulation with CHIMERE. 1. In :bash:`datavect`, define the :bash:`component` containing the original files, with the usual information on how to interpolate them onto the CTM grid. In the following example, two raw emission sources are used for two methane-emitting sectors. .. code-block:: yaml datavect: plugin: name: standard version: std components: meteo: plugin: name: CHIMERE version: std type: meteo dir: /home/chimereicos/espigrad_reanalysis/inputs/meteo_from_old/ file: METEO.%Y%m%d%H.24.nc file_freq: 1D flux: parameter: CO2: plugin: name: CHIMERE version: AEMISSIONS type: flux dir: /home/chimereicos/espigrad_reanalysis/inputs/emis_from_independent/ file: AEMISSIONS.2019%m28%H.24.nc file_freq: 1D sectorflux: parameters: CH4emisS10: plugin: name: CHIMERE version: AEMISSIONS type: flux dir: /home/chimereicos/espigrad_reanalysis/inputs/emis_from_independent/ file: LNAEMISSIONS.2019%m28%H.24.nc file_freq: 1D varname: LOGSNAP10 CH4emisS9: plugin: name: EDGAR version: v5 type: flux dir: /tmp/PYCIF_DATA_TEST/RAW/EMISSIONS/EDGARV5/TOTAL regrid: method: mass-conservation time_interpolation: method: linear unit_conversion: scale: 1.368e+21 vertical_interpolation: method: closest fill_nans: false fill_nans_value: 0 varname: emi_ch4_snap9 file: 'v50_N2O_%Y.0.1x0.1.nc' Note that: - in the example, CO2 uses ready-made files, while CH4 must be built from the :bash:`sectorflux` component. - component names can be freely chosen by the user - parameter names can be freely chosen by the user 2. In :bash:`controlvect`, add the :bash:`transform_pipe` that defines the transformations to apply. .. code-block:: yaml controlvect: plugin: name: standard version: std type: controlvect save_out_netcdf: True transform_pipe: TBC 3. Use the elementary transformations to specify the chain of custom transformations that convert :bash:`datavect` components into the actual CTM inputs. .. code-block:: yaml controlvect: plugin: name: standard version: std type: controlvect save_out_netcdf: True transform_pipe: take_exp: # take the exponential of emissions provided as ln plugin: name: exp type: transform version: std component: sectorflux parameters_in: - CH4emisS10 take_sum_emis: # sum various sectors to obtain the total plugin: name: families version: std type: transform component: sectorflux parameters_in: [ CH4emisS10, CH4emisS9] component_out: flux parameter_out: CH4 .. note:: The CIF applies transformations in the order they appear in the YAML file. In the example, CH4emisS10 is exponentiated **first**, then the result is summed with CH4emisS9. Note that: - transformation names are freely chosen by the user (use descriptive names as a best practice) - the arguments for each transformation type are documented :doc:`here` - :bash:`component` specifies the input component for the transformation, and must match a name defined in :bash:`datavect` - after applying the :bash:`transform_pipe`, the resulting component names must match the required CTM input names; therefore, the final :bash:`component_out` in a chain must match the component name expected by the CTM 4. Run XX with :bash:`only_init`?XX and check with the :doc:`files for checks` that the personalized transforms in controlvect do what is expected. 5. Run the full forward.XX wchich checks?XX For combining the outputs of the CTM ..................................... We take here the example of isotopic data for a forward simulation. 1. In :bash:`datavect`, define the :bash:`component` containing the observation files to compare with the simulation. In the following example, isotopic methane data are used. They are already formatted as a CIF input monitor file. .. code-block:: yaml datavect: plugin: name: standard version: std components: concs: dir: ${PYCIF_DATATEST}/LMDZ//clumped_isotopes//monitor/ file: monitor_filtered_rsd_new.nc file_statlev: ${PYCIF_DATATEST}/LMDZ//simple/stations_levels_LMDZ39lev.txt parameters: CH4: {} d13C: {} Note that :XXXX what are the parameters???XXX 2. In :bash:`obsvect`, add the :bash:`transform_pipe` that defines the transformations to apply. .. code-block:: yaml obsvect: plugin: name: standard version: std dump: True transform_pipe: TBC 3. Use the elementary transformations to specify the chain of custom transformations that produce observation-equivalent values from CTM outputs. .. code-block:: yaml obsvect: plugin: name: standard version: std dump: True transform_pipe: makeiso: plugin: name: conc2ratio type: transform version: std component: - concs - concs XXXXXXXXXXXXXXXXXXXXXXXXX parameters_in: names: - 12CH4 - 13CH4 parameters_out: d13C: isotopologues: - 13CH4 refs: - 12CH4 standard: 0.0112372 iso_mass: - 16.031 - 17.035 names: - d13C - CH4 spec_mass: 16.0425 unit: volume