Downloading and preprocessing EDGAR dataset

Downloading and preprocessing EDGAR dataset#

GCP-CH4 uses EDGAR CH4 fluxes inventory as for its inversions’ anthropogenic fluxes a priori. This inventory can be downloaded from the EDGAR webiste.

The script bellow is an example for downloading and unzipping the EDGARv6.0 CH4 dataset:

Show/Hide download_EDGARv6.0.sh

 1#!/usr/bin/bash
 2
 3DIR="$(cd $(dirname $0); pwd)"
 4
 5year_start=2000
 6year_stop=2018
 7
 8base_url="https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/EDGAR/datasets/v60_GHG/CH4"
 9
10sector_lists=( \
11    'AGS' 'AWB' 'CHE' 'ENE' 'ENF' 'FFF' 'IND' 'IRO' 'MNM' \
12    'PRO_COAL' 'PRO_GAS' 'PRO_OIL' 'RCO' 'REF_TRF' 'SWD_INC' 'SWD_LDF' \
13    'TNR_Aviation_CDS' 'TNR_Aviation_CRS' 'TNR_Aviation_LTO' 'TNR_Aviation_SPS' \
14    'TNR_Other' 'TNR_Ship' 'TRO_noRES' 'WWT' \
15)
16
17for sector in "${sector_list[@]}"; do
18    echo "Downloading Sector ${sector}"
19    mkdir -p "${DIR}/${sector}"
20
21    if [ "${sector}" = "TNR_Aviation_SPS" ]; then
22        if [ $year_stop -gt 2003 ];then
23            sector_year_stop=2003
24        fi
25    else
26        sector_year_stop=$year_stop
27    fi
28
29    for year in $(seq ${year_start} ${sector_year_stop}); do
30          zip_file="${DIR}/${sector}/CH4_${sector}_${year}.zip"
31          wget "${base_url}/${sector}/v6.0_CH4_${year}_monthly_${sector}_nc.zip" \
32              --output-document "${zip_file}"
33          unzip -o "${zip_file}" -d "${DIR}/${sector}"
34          rm -vf "${zip_file}"
35
36          sleep 1 # Can prevent "connection denied" errors
37        fi
38    done
39done

EDGAR fluxes are divided in numerous sectors and use a \(0.1° \times 0.1°\) grid. GCP-CH4 a priori fluxes use a coarser grid and fewer sectors. The YAML files below groups and regrids EDGAR’s fluxes files to match the GCP-CH4 a priori format.

With those example the files are written in the folder /tmp/GCP_Fluxes.

Warning

The YAML file gcp-regrid-and-group-edgar-full.yml is not automatically tested, but the second example file gcp-regrid-and-group-edgar.yml is.

Note

The “TNR_Aviation_SPS” sector ends at the end of year 2003, to compute the fluxes between 2004 and 2018 remove the two appearances of this sector in the YAML file (one in controlvect/transfrom_pipe/sum_trans and one in datavect/components/flux/parameters) and modify the datei and datef accordingly.

^ *
  1#####################
  2# PYCIF config file #
  3#####################
  4
  5###############################################################################
  6#
  7# Regrid and sum by categories EDGARv6.0 CH4 monthly fluxe files
  8#
  9###############################################################################
 10# Paths
 11
 12# Outputs directory
 13rootdir: &rootdir /tmp/cif-gcp
 14
 15# EDGAR dataset directory
 16edgardir: &edgardir /path/to/edgar/download/directory
 17
 18# Target domain file directory
 19domaindir: &domaindir /path/to/domain/directory
 20
 21# Target domain filename
 22domainfile: &domainfile domain_1x1.nc
 23
 24###############################################################################
 25# PYCIF parameters
 26
 27verbose: 1
 28
 29outdir: !join &outdir [*rootdir, /GCP_Fluxes]
 30workdir: !join [*rootdir, /workdir]
 31logfile: pycif.log
 32
 33datei: 2000-01-01
 34datef: 2004-01-01
 35
 36###############################################################################
 37
 38platform:
 39  plugin:
 40    name: docker
 41    version: cif
 42    
 43###############################################################################
 44
 45obsoperator:
 46  plugin:
 47    name: standard
 48    version: std
 49
 50  init_inputs:
 51    components:
 52      flux: []
 53  ignore_model: true
 54
 55###############################################################################
 56
 57obsvect:
 58  plugin:
 59    name: standard
 60    version: std
 61
 62###############################################################################
 63
 64domain:
 65  plugin:
 66    name: gridded_netcdf
 67    version: std
 68  dir: *domaindir
 69  file: *domainfile
 70
 71###############################################################################
 72
 73controlvect:
 74  plugin:
 75    name: standard
 76    version: std
 77
 78  transform_pipe:
 79    sum_live:
 80      plugin:
 81        name: families
 82        version: std
 83        type: transform
 84      component: flux
 85      parameters_in:
 86        - ENF
 87        - MNM
 88      parameter_out: live
 89
 90    sum_waste:
 91      plugin:
 92        name: families
 93        version: std
 94        type: transform
 95      component: flux
 96      parameters_in:
 97        - SWD_LDF
 98        - SWD_INC
 99        - WWT
100      parameter_out: waste
101
102    sum_oilgas:
103      plugin:
104        name: families
105        version: std
106        type: transform
107      component: flux
108      parameters_in:
109        - PRO_OIL
110        - PRO_GAS
111        - FFF
112      parameter_out: oilgas
113
114    sum_indus:
115      plugin:
116        name: families
117        version: std
118        type: transform
119      component: flux
120      parameters_in:
121        - CHE
122        - IRO
123        - ENE
124        - IND
125        - REF_TRF
126      parameter_out: indus
127
128    sum_trans:
129      plugin:
130        name: families
131        version: std
132        type: transform
133      component: flux
134      parameters_in:
135        - TRO_noRES
136        - TNR_Aviation_CDS
137        - TNR_Aviation_CRS
138        - TNR_Aviation_LTO
139        - TNR_Aviation_SPS
140        - TNR_Other
141        - TNR_Ship
142      parameter_out: trans
143
144    dump_live:
145      plugin:
146        name: dump2format
147        version: std
148        type: transform
149      component: flux
150      parameter: live
151      dump_dir: !join [*outdir, /live]
152      dump_file: fch4.live.gcp2022.1x1.%Y.nc
153      dump_format:
154        type: flux
155        name: gridded_netcdf
156        version: std
157      use_defined_domain: True
158      resample:
159        file_freq: 1YS
160        var_freq: 1MS
161
162    dump_rice:
163      plugin:
164        name: dump2format
165        version: std
166        type: transform
167      component: flux
168      parameter: AGS
169      dump_dir: !join [*outdir, /rice]
170      dump_file: fch4.rice.gcp2022.1x1.%Y.nc
171      dump_format:
172        type: flux
173        name: gridded_netcdf
174        version: std
175      use_defined_domain: True
176      resample:
177        file_freq: 1YS
178        var_freq: 1MS
179
180    dump_waste:
181      plugin:
182        name: dump2format
183        version: std
184        type: transform
185      component: flux
186      parameter: waste
187      dump_dir: !join [*outdir, /waste]
188      dump_file: fch4.waste.gcp2022.1x1.%Y.nc
189      dump_format:
190        type: flux
191        name: gridded_netcdf
192        version: std
193      use_defined_domain: True
194      resample:
195        file_freq: 1YS
196        var_freq: 1MS
197
198    dump_oilgas:
199      plugin:
200        name: dump2format
201        version: std
202        type: transform
203      component: flux
204      parameter: oilgas
205      dump_dir: !join [*outdir, /oilgas]
206      dump_file: fch4.oilgas.gcp2022.1x1.%Y.nc
207      dump_format:
208        type: flux
209        name: gridded_netcdf
210        version: std
211      use_defined_domain: True
212      resample:
213        file_freq: 1YS
214        var_freq: 1MS
215
216    dump_coal:
217      plugin:
218        name: dump2format
219        version: std
220        type: transform
221      component: flux
222      parameter: PRO_COAL
223      dump_dir: !join [*outdir, /coal]
224      dump_file: fch4.coal.gcp2022.1x1.%Y.nc
225      dump_format:
226        type: flux
227        name: gridded_netcdf
228        version: std
229      use_defined_domain: True
230      resample:
231        file_freq: 1YS
232        var_freq: 1MS
233
234    dump_indus:
235      plugin:
236        name: dump2format
237        version: std
238        type: transform
239      component: flux
240      parameter: indus
241      dump_dir: !join [*outdir, /indus]
242      dump_file: fch4.indus.gcp2022.1x1.%Y.nc
243      dump_format:
244        type: flux
245        name: gridded_netcdf
246        version: std
247      use_defined_domain: True
248      resample:
249        file_freq: 1YS
250        var_freq: 1MS
251
252    dump_trans:
253      plugin:
254        name: dump2format
255        version: std
256        type: transform
257      component: flux
258      parameter: trans
259      dump_dir: !join [*outdir, /trans]
260      dump_file: fch4.trans.gcp2022.1x1.%Y.nc
261      dump_format:
262        type: flux
263        name: gridded_netcdf
264        version: std
265      use_defined_domain: True
266      resample:
267        file_freq: 1YS
268        var_freq: 1MS
269
270    dump_biof:
271      plugin:
272        name: dump2format
273        version: std
274        type: transform
275      component: flux
276      parameter: RCO
277      dump_dir: !join [*outdir, /biof]
278      dump_file: fch4.biof.gcp2022.1x1.%Y.nc
279      dump_format:
280        type: flux
281        name: gridded_netcdf
282        version: std
283      use_defined_domain: True
284      resample:
285        file_freq: 1YS
286        var_freq: 1MS
287      
288
289###############################################################################
290
291datavect:
292  plugin:
293    name: standard
294    version: std
295
296  components:
297    flux:
298      dir: *edgardir
299      file_freq: 1MS
300      varname: emi_ch4
301
302      regrid:
303        method: mass-conservation
304
305      parameters:
306        AGS:
307          plugin:
308            name: gridded_netcdf
309            version: std
310            type: flux
311          file: AGS/v6.0_CH4_%Y_%-m_AGS.0.1x0.1.nc
312          add_time_coord: True
313          var_freq: 1M
314          lon_min: 0.0
315          lon_max: 360.0
316          delta_lon: 0.1
317          delta_lat: 0.1
318
319        CHE:
320          plugin:
321            name: gridded_netcdf
322            version: std
323            type: flux
324          file: CHE/v6.0_CH4_%Y_%-m_CHE.0.1x0.1.nc
325          add_time_coord: True
326          var_freq: 1M
327          lon_min: 0.0
328          lon_max: 360.0
329          delta_lon: 0.1
330          delta_lat: 0.1
331
332        ENE:
333          plugin:
334            name: gridded_netcdf
335            version: std
336            type: flux
337          file: ENE/v6.0_CH4_%Y_%-m_ENE.0.1x0.1.nc
338          add_time_coord: True
339          var_freq: 1M
340          lon_min: 0.0
341          lon_max: 360.0
342          delta_lon: 0.1
343          delta_lat: 0.1
344
345        ENF:
346          plugin:
347            name: gridded_netcdf
348            version: std
349            type: flux
350          file: ENF/v6.0_CH4_%Y_%-m_ENF.0.1x0.1.nc
351          add_time_coord: True
352          var_freq: 1M
353          lon_min: 0.0
354          lon_max: 360.0
355          delta_lon: 0.1
356          delta_lat: 0.1
357
358        FFF:
359          plugin:
360            name: gridded_netcdf
361            version: std
362            type: flux
363          file: FFF/v6.0_CH4_%Y_%-m_FFF.0.1x0.1.nc
364          add_time_coord: True
365          var_freq: 1M
366          lon_min: 0.0
367          lon_max: 360.0
368          delta_lon: 0.1
369          delta_lat: 0.1
370
371        IND:
372          plugin:
373            name: gridded_netcdf
374            version: std
375            type: flux
376          file: IND/v6.0_CH4_%Y_%-m_IND.0.1x0.1.nc
377          add_time_coord: True
378          var_freq: 1M
379          lon_min: 0.0
380          lon_max: 360.0
381          delta_lon: 0.1
382          delta_lat: 0.1
383
384        IRO:
385          plugin:
386            name: gridded_netcdf
387            version: std
388            type: flux
389          file: IRO/v6.0_CH4_%Y_%-m_IRO.0.1x0.1.nc
390          add_time_coord: True
391          var_freq: 1M
392          lon_min: 0.0
393          lon_max: 360.0
394          delta_lon: 0.1
395          delta_lat: 0.1
396
397        MNM:
398          plugin:
399            name: gridded_netcdf
400            version: std
401            type: flux
402          file: MNM/v6.0_CH4_%Y_%-m_MNM.0.1x0.1.nc
403          add_time_coord: True
404          var_freq: 1M
405          lon_min: 0.0
406          lon_max: 360.0
407          delta_lon: 0.1
408          delta_lat: 0.1
409
410        PRO_COAL:
411          plugin:
412            name: gridded_netcdf
413            version: std
414            type: flux
415          file: PRO_COAL/v6.0_CH4_%Y_%-m_PRO_COAL.0.1x0.1.nc
416          add_time_coord: True
417          var_freq: 1M
418          lon_min: 0.0
419          lon_max: 360.0
420          delta_lon: 0.1
421          delta_lat: 0.1
422
423        PRO_GAS:
424          plugin:
425            name: gridded_netcdf
426            version: std
427            type: flux
428          file: PRO_GAS/v6.0_CH4_%Y_%-m_PRO_GAS.0.1x0.1.nc
429          add_time_coord: True
430          var_freq: 1M
431          lon_min: 0.0
432          lon_max: 360.0
433          delta_lon: 0.1
434          delta_lat: 0.1
435
436        PRO_OIL:
437          plugin:
438            name: gridded_netcdf
439            version: std
440            type: flux
441          file: PRO_OIL/v6.0_CH4_%Y_%-m_PRO_OIL.0.1x0.1.nc
442          add_time_coord: True
443          var_freq: 1M
444          lon_min: 0.0
445          lon_max: 360.0
446          delta_lon: 0.1
447          delta_lat: 0.1
448
449        RCO:
450          plugin:
451            name: gridded_netcdf
452            version: std
453            type: flux
454          file: RCO/v6.0_CH4_%Y_%-m_RCO.0.1x0.1.nc
455          add_time_coord: True
456          var_freq: 1M
457          lon_min: 0.0
458          lon_max: 360.0
459          delta_lon: 0.1
460          delta_lat: 0.1
461
462        REF_TRF:
463          plugin:
464            name: gridded_netcdf
465            version: std
466            type: flux
467          file: REF_TRF/v6.0_CH4_%Y_%-m_REF_TRF.0.1x0.1.nc
468          add_time_coord: True
469          var_freq: 1M
470          lon_min: 0.0
471          lon_max: 360.0
472          delta_lon: 0.1
473          delta_lat: 0.1
474
475        SWD_INC:
476          plugin:
477            name: gridded_netcdf
478            version: std
479            type: flux
480          file: SWD_INC/v6.0_CH4_%Y_%-m_SWD_INC.0.1x0.1.nc
481          add_time_coord: True
482          var_freq: 1M
483          lon_min: 0.0
484          lon_max: 360.0
485          delta_lon: 0.1
486          delta_lat: 0.1
487
488        SWD_LDF:
489          plugin:
490            name: gridded_netcdf
491            version: std
492            type: flux
493          file: SWD_LDF/v6.0_CH4_%Y_%-m_SWD_LDF.0.1x0.1.nc
494          add_time_coord: True
495          var_freq: 1M
496          lon_min: 0.0
497          lon_max: 360.0
498          delta_lon: 0.1
499          delta_lat: 0.1
500
501        TNR_Aviation_CDS:
502          plugin:
503            name: gridded_netcdf
504            version: std
505            type: flux
506          file: TNR_Aviation_CDS/v6.0_CH4_%Y_%-m_TNR_Aviation_CDS.0.1x0.1.nc
507          add_time_coord: True
508          var_freq: 1M
509          lon_min: 0.0
510          lon_max: 360.0
511          delta_lon: 0.1
512          delta_lat: 0.1
513
514        TNR_Aviation_CRS:
515          plugin:
516            name: gridded_netcdf
517            version: std
518            type: flux
519          file: TNR_Aviation_CRS/v6.0_CH4_%Y_%-m_TNR_Aviation_CRS.0.1x0.1.nc
520          add_time_coord: True
521          var_freq: 1M
522          lon_min: 0.0
523          lon_max: 360.0
524          delta_lon: 0.1
525          delta_lat: 0.1
526
527        TNR_Aviation_LTO:
528          plugin:
529            name: gridded_netcdf
530            version: std
531            type: flux
532          file: TNR_Aviation_LTO/v6.0_CH4_%Y_%-m_TNR_Aviation_LTO.0.1x0.1.nc
533          add_time_coord: True
534          var_freq: 1M
535          lon_min: 0.0
536          lon_max: 360.0
537          delta_lon: 0.1
538          delta_lat: 0.1
539
540        TNR_Aviation_SPS:
541          plugin:
542            name: gridded_netcdf
543            version: std
544            type: flux
545          file: TNR_Aviation_SPS/v6.0_CH4_%Y_%-m_TNR_Aviation_SPS.0.1x0.1.nc
546          add_time_coord: True
547          var_freq: 1M
548          lon_min: 0.0
549          lon_max: 360.0
550          delta_lon: 0.1
551          delta_lat: 0.1
552
553        TNR_Other:
554          plugin:
555            name: gridded_netcdf
556            version: std
557            type: flux
558          file: TNR_Other/v6.0_CH4_%Y_%-m_TNR_Other.0.1x0.1.nc
559          add_time_coord: True
560          var_freq: 1M
561          lon_min: 0.0
562          lon_max: 360.0
563          delta_lon: 0.1
564          delta_lat: 0.1
565
566        TNR_Ship:
567          plugin:
568            name: gridded_netcdf
569            version: std
570            type: flux
571          file: TNR_Ship/v6.0_CH4_%Y_%-m_TNR_Ship.0.1x0.1.nc
572          add_time_coord: True
573          var_freq: 1M
574          lon_min: 0.0
575          lon_max: 360.0
576          delta_lon: 0.1
577          delta_lat: 0.1
578
579        TRO_noRES:
580          plugin:
581            name: gridded_netcdf
582            version: std
583            type: flux
584          file: TRO_noRES/v6.0_CH4_%Y_%-m_TRO_noRES.0.1x0.1.nc
585          add_time_coord: True
586          var_freq: 1M
587          lon_min: 0.0
588          lon_max: 360.0
589          delta_lon: 0.1
590          delta_lat: 0.1
591
592        WWT:
593          plugin:
594            name: gridded_netcdf
595            version: std
596            type: flux
597          file: WWT/v6.0_CH4_%Y_%-m_WWT.0.1x0.1.nc
598          add_time_coord: True
599          var_freq: 1M
600          lon_min: 0.0
601          lon_max: 360.0
602          delta_lon: 0.1
603          delta_lat: 0.1

Show/Hide YAML file gcp-regrid-and-group-edgar.yml (automatically tested)

TODO !!

The groups are:

  • livestocks

    • ENF

    • MNM

  • rice

    • AGS

  • waste

    • SWD_LDF

    • SWD_INC

    • WWT

  • oil & gas

    • PRO_OIL

    • PRO_GAS

    • FFF

  • coal

    • PRO_COAL

  • industry

    • CHE

    • IRO

    • ENE

    • IND

    • REF_TRF

  • transports

    • TRO_noRES

    • TNR_Aviation_CDS

    • TNR_Aviation_CRS

    • TNR_Aviation_LTO

    • TNR_Aviation_SPS

    • TNR_Other

    • TNR_Ship

Here is the target domain NetCDF file which contains the coordinates for a \(1° \times 1°\) grid.

Show/Hide domain_1x1.nc

netcdf domain_1x1 {
dimensions:
    lat = 180 ;
    lon = 360 ;
    bounds = 2 ;
variables:
    double lat(lat) ;
        lat:_FillValue = NaN ;
        lat:long_name = "latitude" ;
        lat:standard_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:bounds = "lat_bounds" ;
    double lon(lon) ;
        lon:_FillValue = NaN ;
        lon:long_name = "longitude" ;
        lon:standard_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:bounds = "lon_bounds" ;
    double lat_bounds(lat, bounds) ;
        lat_bounds:_FillValue = NaN ;
        lat_bounds:long_name = "latitude bounds" ;
        lat_bounds:standard_name = "latitude_bounds" ;
    double lon_bounds(lon, bounds) ;
        lon_bounds:_FillValue = NaN ;
        lon_bounds:long_name = "longitude bounds" ;
        lon_bounds:standard_name = "longitude_bounds" ;

// global attributes:
    :coordinates = "lon_bounds lat_bounds" ;
}