Downloading and preprocessing EDGAR dataset
==================================================

GCP-CH4 uses EDGAR CH4 fluxes inventory as for its inversions' anthropogenic
fluxes a priori.
This inventory can be downloaded from the `EDGAR webiste <https://edgar.jrc.ec.europa.eu/emissions_data_and_maps>`_.

The script bellow is an example for downloading and unzipping the EDGARv6.0 CH4
dataset:

.. container:: toggle

  .. container:: header

    Show/Hide ``download_EDGARv6.0.sh``

  .. code-block:: bash
    :linenos:

    #!/usr/bin/bash

    DIR="$(cd $(dirname $0); pwd)"

    year_start=2000
    year_stop=2018

    base_url="https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/EDGAR/datasets/v60_GHG/CH4"

    sector_lists=( \
        'AGS' 'AWB' 'CHE' 'ENE' 'ENF' 'FFF' 'IND' 'IRO' 'MNM' \
        'PRO_COAL' 'PRO_GAS' 'PRO_OIL' 'RCO' 'REF_TRF' 'SWD_INC' 'SWD_LDF' \
        'TNR_Aviation_CDS' 'TNR_Aviation_CRS' 'TNR_Aviation_LTO' 'TNR_Aviation_SPS' \
        'TNR_Other' 'TNR_Ship' 'TRO_noRES' 'WWT' \
    )

    for sector in "${sector_list[@]}"; do
        echo "Downloading Sector ${sector}"
        mkdir -p "${DIR}/${sector}"

        if [ "${sector}" = "TNR_Aviation_SPS" ]; then
            if [ $year_stop -gt 2003 ];then
                sector_year_stop=2003
            fi
        else
            sector_year_stop=$year_stop
        fi

        for year in $(seq ${year_start} ${sector_year_stop}); do
              zip_file="${DIR}/${sector}/CH4_${sector}_${year}.zip"
              wget "${base_url}/${sector}/v6.0_CH4_${year}_monthly_${sector}_nc.zip" \
                  --output-document "${zip_file}"
              unzip -o "${zip_file}" -d "${DIR}/${sector}"
              rm -vf "${zip_file}"

              sleep 1 # Can prevent "connection denied" errors 
            fi
        done
    done

EDGAR fluxes are divided in numerous sectors and use a 
:math:`0.1° \times 0.1°` grid. 
GCP-CH4 a priori fluxes use a coarser grid and fewer sectors.
The YAML files below groups and regrids EDGAR's fluxes files to match the GCP-CH4
a priori format.

With those example the files are written in the folder ``/tmp/GCP_Fluxes``.

.. warning::
    The YAML file ``gcp-regrid-and-group-edgar-full.yml`` is not automatically
    tested, but the second example file ``gcp-regrid-and-group-edgar.yml`` is.

.. note::
    The "TNR_Aviation_SPS" sector ends at the end of year 2003, to compute the
    fluxes between 2004 and 2018 remove the two appearances of this sector in
    the YAML file (one in ``controlvect/transfrom_pipe/sum_trans`` and one in
    ``datavect/components/flux/parameters``) and modify the ``datei`` and
    ``datef`` accordingly.

.. container:: toggle

  .. container:: header

      Show/Hide YAML file ``gcp-regrid-and-group-edgar-full.yml`` (**not automatically tested**)

  .. literalinclude:: yml/gcp-regrid-and-group-edgar-full.yml
      :linenos:
      :language: yaml

.. container:: toggle

  .. container:: header

      Show/Hide YAML file ``gcp-regrid-and-group-edgar.yml`` (automatically tested)

  TODO !!

The groups are:
  
  - livestocks
  
    * ENF
    * MNM
  
  - rice
  
    * AGS
  
  - waste
   
    * SWD_LDF
    * SWD_INC
    * WWT
 
  - oil & gas
   
    * PRO_OIL
    * PRO_GAS
    * FFF
  
  - coal
   
    * PRO_COAL
 
  - industry
    
    * CHE
    * IRO
    * ENE
    * IND
    * REF_TRF
 
  - transports
    
    * TRO_noRES
    * TNR_Aviation_CDS
    * TNR_Aviation_CRS
    * TNR_Aviation_LTO
    * TNR_Aviation_SPS
    * TNR_Other
    * TNR_Ship


Here is the target domain NetCDF file which contains the coordinates for a
:math:`1° \times 1°` grid.


.. container:: toggle

  .. container:: header

      Show/Hide ``domain_1x1.nc``

  .. code-block:: c

    netcdf domain_1x1 {
    dimensions:
        lat = 180 ;
        lon = 360 ;
        bounds = 2 ;
    variables:
        double lat(lat) ;
            lat:_FillValue = NaN ;
            lat:long_name = "latitude" ;
            lat:standard_name = "latitude" ;
            lat:units = "degrees_north" ;
            lat:bounds = "lat_bounds" ;
        double lon(lon) ;
            lon:_FillValue = NaN ;
            lon:long_name = "longitude" ;
            lon:standard_name = "longitude" ;
            lon:units = "degrees_east" ;
            lon:bounds = "lon_bounds" ;
        double lat_bounds(lat, bounds) ;
            lat_bounds:_FillValue = NaN ;
            lat_bounds:long_name = "latitude bounds" ;
            lat_bounds:standard_name = "latitude_bounds" ;
        double lon_bounds(lon, bounds) ;
            lon_bounds:_FillValue = NaN ;
            lon_bounds:long_name = "longitude bounds" ;
            lon_bounds:standard_name = "longitude_bounds" ;

    // global attributes:
        :coordinates = "lon_bounds lat_bounds" ;
    }