pycif.plugins.datastreams.fluxes.gridded_NetCDF — API reference

Configuration reference: gridded_NetCDF plugin

pycif.plugins.datastreams.fluxes.gridded_NetCDF.fetch.fetch(ref_dir, ref_file, input_interval, target_dir, tracer=None, component=None, **kwargs)

Fetch gridded NetCDF flux files and derive the validity time intervals they cover.

Depending on tracer.is_climatology and tracer.file_freq, builds the list of candidate file dates, links the matching files into target_dir, and inspects the time coordinate of each file (decoding it, shifting years to align with the requested period, and expanding leap years) to compute the exact [start, end] validity interval of every time record that overlaps input_interval.

Parameters:

• ref_dir (str) – Directory containing the reference input files.

• ref_file (str) – Filename pattern of the input files (a strftime format string), or the exact file name for climatology data.

• input_interval (list[datetime.datetime]) – [date_i, date_f] simulation sub-period to cover.

• target_dir (str) – Directory where the resolved files are linked.

• tracer – The flux tracer plugin, providing file_freq, is_climatology, time_midpoint, time_endpoint, add_time_coord and other time-related input arguments.

• component – Unused, kept for interface consistency with other fetch functions.

Returns:

(list_files, list_dates), each keyed by the requested sub-period date, mapping to the list of resolved file paths and the list of [start, end] date-interval pairs found in those files.

Return type:

tuple[dict, dict]

Raises:

• IllegalArgumentError – If incompatible combinations of time-related arguments are used (e.g. time_unit with time_format, or file_freq with is_climatology).

• CifError – If a resolved file does not exist on disk, or if the NetCDF file cannot be opened due to ambiguous dimension/variable naming.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.get_domain.get_domain(ref_dir, ref_file, input_interval=None, target_dir=None, tracer=None)

Read information from the reference file to define the data horizontal and, if relevant, vertical domain.

Parameters:

• ref_dir (str) – the path to the input files

• ref_file (str) – format of the input files

• input_interval (list) – simulation interval (start and end dates)

• target_dir (str) – where to copy

• tracer – the tracer Plugin

Returns:

a domain class object, with the definition of the center grid cells coordinates, as well as corners

Return type:

domain (Domain)

pycif.plugins.datastreams.fluxes.gridded_NetCDF.read.read(self, name, varnames, dates, files, interpol_flx=False, tracer=None, model=None, ddi=None, debug_read=False, **kwargs)

Get fluxes from raw gridded NetCDF files and load them into a pyCIF variable.

For each requested (date_i, date_f)/file pair, opens the file (only re-opening when the file path changes), locates and decodes its time coordinate, shifts years and expands leap years as needed, slices the exact requested time interval, optionally slices/reduces extra dimensions (tracer.slice_dimension), renames the lat/lon/time/vertical dimensions to the pyCIF convention (time, lev, lat, lon), and checks the resulting shape against the domain.

Parameters:

• self – The flux tracer plugin instance, providing is_climatology, vertical_dim_name, sort_lat, sort_lon, sum_along_dim and domain.

• name (str) – name of the component

• varnames (list[str]) – original names of variables to read; use name if varnames is empty

• dates (list) – list of the date intervals to extract

• files (list) – list of the files matching dates

• interpol_flx (bool) – Unused directly, kept for interface consistency with other flux plugins.

• tracer – The flux tracer plugin, providing the same time-related input arguments used in fetch, plus group_name, drop_variables and slice_dimension.

• model – Unused directly, kept for interface consistency.

• ddi – Unused directly, kept for interface consistency.

• debug_read (bool) – Unused directly, kept for interface consistency.

Returns:

the actual data with dimension:

time, levels, latitudes, longitudes

Return type:

xr.DataArray

Raises:

• CifError – If a file cannot be opened due to ambiguous dimension/variable naming, or if an unknown slice_dimension method is requested.

• CifKeyError – If a requested variable is missing from a file, or if the resulting dimensions do not match the expected ('time', 'lev', 'lat', 'lon') set.

• CifValueError – If a time slice yields zero or multiple matches, or if the data shape is inconsistent with the domain.

• CifTypeError – If multiple varnames are given without tracer.sum_variables set.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.find_time_coord(ds: Dataset, tracer, ref_date)

Locate, or synthesize, the time coordinate of a dataset.

If tracer.add_time_coord is set, a 'time' coordinate is created from scratch (a single value equal to ref_date, or a tracer.var_freq-spaced range starting at ref_date when the time dimension has more than one element) and assigned to the dataset. Otherwise, the existing time coordinate is looked up, either by its explicit name (tracer.time_varname) or by its standard_name/ long_name/'time' attributes.

Parameters:

• ds (xr.Dataset) – Dataset to inspect (or, in the add_time_coord case, modify).

• tracer – The flux tracer plugin, providing add_time_coord, time_dimname, var_freq and time_varname.

• ref_date – Reference date used as the first timestamp when synthesizing a time coordinate.

Returns:

The (possibly modified) dataset and the name of its time coordinate/variable.

Return type:

tuple[xr.Dataset, str]

Raises:

CifValueError – If add_time_coord is set without var_freq while the time dimension has more than one element, or if no time coordinate can be found in the dataset.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.get_calendar(ds: Dataset, time_varname: str)

Retrieve the calendar used by a dataset’s time variable.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• time_varname (str) – Name of the time variable.

Returns:

The calendar name (e.g. 'gregorian', 'noleap'), taken from the calendar attribute if present, otherwise from the first decoded time value; None if unavailable.

Return type:

str or None

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.preprocess_time_coord(ds: Dataset, tracer, time_varname: str, ref_date, time_midpoint: bool = False, time_endpoint: bool = False, file_freq: str = '', is_climatology: bool = False) → Dataset

Align, decode, and optionally shift a dataset’s time coordinate.

Renames the time dimension to match the time variable name, decodes the time values to datetime64 if needed (or if tracer.force_var_freq is set), and, if time_midpoint or time_endpoint is set, shifts all timestamps back to the beginning of their validity period (by half or a full time step, respectively).

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• tracer – The flux tracer plugin, forwarded to decode_datetimes.

• time_varname (str) – Name of the time variable/coordinate.

• ref_date – Reference date, forwarded to decode_datetimes.

• time_midpoint (bool) – Whether the time values mark the midpoint of each validity period.

• time_endpoint (bool) – Whether the time values mark the end of each validity period.

• file_freq (str) – File cadence, forwarded to decode_datetimes.

• is_climatology (bool) – Whether the data is climatological, forwarded to decode_datetimes.

Returns:

The dataset with a decoded and, if requested, shifted time coordinate.

Return type:

xr.Dataset

Raises:

CifError – If period shifting is requested but the time step is not uniform.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.decode_datetimes(ds: Dataset, tracer, var_name: str, ref_date, file_freq: str = '', is_climatology: bool = False) → Dataset

Decode a non-datetime time variable into datetime64 values.

Uses tracer.time_unit (a NetCDF-style time unit, possibly embedding a date format) or tracer.time_format (a strftime format) to decode the raw values, then, if tracer.var_freq is set, optionally overrides the decoded values with a regular date range starting at the file’s nearest file_freq boundary on-or-before ref_date.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• tracer – The flux tracer plugin, providing time_unit, time_format, var_freq and force_var_freq.

• var_name (str) – Name of the time variable to decode.

• ref_date – Reference date used to resolve the time unit/format and as an anchor when imposing var_freq.

• file_freq (str) – File cadence, used when imposing var_freq.

• is_climatology (bool) – Whether the data is climatological; used to sanity-check that the decoded time length is compatible with var_freq.

Returns:

The dataset with var_name decoded to datetime64.

Return type:

xr.Dataset

Raises:

• CifError – If file_freq is empty while needed to impose var_freq.

• CifValueError – If the decoded time coordinate is still not datetime64, or if the time length is incompatible with a climatology’s var_freq.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.decode_datetimes_with_units(ds: Dataset, var_name: str, units: str) → Dataset

Decode datetimes from a CF-style "<unit> since <date>" units string.

Supports years, months, and years since <date> units, as well as an "as <format>" convention that delegates to decode_datetimes_with_format.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• var_name (str) – Name of the time variable to decode.

• units (str) – The units string to interpret.

Returns:

The dataset with var_name decoded to datetime64.

Return type:

xr.Dataset

Raises:

CifValueError – If units does not match a supported pattern.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.decode_datetimes_with_format(ds: Dataset, var_name: str, time_format: str) → Dataset

Decode datetimes from a variable using a strftime format string.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• var_name (str) – Name of the time variable to decode.

• time_format (str) – strftime format used to parse the values, passed to pandas.to_datetime.

Returns:

The dataset with var_name decoded to datetime64.

Return type:

xr.Dataset

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.shift_years(ds: Dataset, var_name: str, year_offset: int) → Dataset

Shift all values of a time variable by a number of years.

Duplicate days that can result from shifting a leap year onto a non-leap year are dropped.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• var_name (str) – Name of the time variable to shift.

• year_offset (int) – Number of years to add (may be negative); a no-op if 0.

Returns:

The dataset with var_name shifted (and duplicate days dropped along its dimension).

Return type:

xr.Dataset

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.convert_calendar(ds: Dataset, tracer, var_name: str, calendar: str) → Dataset

Convert a dataset’s time coordinate to the standard (Gregorian) calendar.

For noleap/365_day calendars, validates that tracer.var_freq (if set) is compatible (daily or sub-daily), then delegates to expand_leap_years to insert the missing February 29th records.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• tracer – The flux tracer plugin, providing var_freq.

• var_name (str) – Name of the time variable.

• calendar (str) – Calendar of the input time coordinate (e.g. 'noleap', '365_day', 'gregorian').

Returns:

The dataset with leap years expanded if needed.

Return type:

xr.Dataset

Raises:

IllegalArgumentError – If calendar is noleap/365_day and tracer.var_freq is set to an incompatible frequency.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.time_coord.expand_leap_years(ds: Dataset, tracer, var_name: str) → Dataset

Insert a synthetic February 29th record for leap years missing it.

For each leap year present in the time coordinate that has a February 28th record but no February 29th one, duplicates the February 28th record (and the associated tracer.period_varname interval, if present, shifted by one day) to stand in for February 29th.

Parameters:

• ds (xr.Dataset) – Dataset holding the time variable.

• tracer – The flux tracer plugin; if it has a period_varname attribute, the corresponding period variable is expanded too.

• var_name (str) – Name of the time variable/dimension to expand.

Returns:

The dataset with a February 29th record inserted for every leap year that was missing one; unchanged if none was missing.

Return type:

xr.Dataset

pycif.plugins.datastreams.fluxes.gridded_NetCDF.utils.get_time_intervals(ds, tracer, time_varname, date_i, date_f, calendar, time_midpoint=False, time_endpoint=False)

Compute the list of validity time intervals covered by a file.

If tracer.period_varname is set, the intervals are read directly from that variable. Otherwise, they are derived from the time coordinate by converting it to Pandas periods (using tracer.var_freq, or falling back to tracer.file_freq for single-timestep files), and filtered down to those overlapping [date_i, date_f].

Parameters:

• ds (xr.Dataset) – Dataset holding the time coordinate (and, if relevant, the period variable).

• tracer – The flux tracer plugin, providing var_freq, file_freq and, optionally, period_varname.

• time_varname (str) – Name of the time coordinate.

• date_i (datetime.datetime) – Start of the simulation window.

• date_f (datetime.datetime) – End of the simulation window.

• calendar (str) – Calendar of the time coordinate (unused directly, kept for interface consistency).

• time_midpoint (bool) – Unused directly, kept for interface consistency.

• time_endpoint (bool) – Unused directly, kept for interface consistency.

Returns:

A list of [start, end] datetime.datetime pairs overlapping the simulation window.

Return type:

list

Raises:

• CifError – If var_freq ends with the 'S' (start) anchor, which to_period does not accept.

• CifValueError – If the time coordinate cannot be parsed/converted to periods, if a single-period file does not overlap the simulation window, or if no period overlaps the simulation window.

pycif.plugins.datastreams.fluxes.gridded_NetCDF.utils.get_year_offset(ds: Dataset, time_varname: str, ref_year: int, is_climatology: bool = False) → int

Compute the year shift needed to align a file’s time coordinate with a target year.

For climatology data, the offset simply maps the file’s (single) year onto ref_year. Otherwise, the file is only shifted if ref_year falls entirely outside the range of years present in the file (shifting to the nearest edge); no shift is applied if ref_year is already covered.

Parameters:

• ds (xr.Dataset) – Dataset holding the time coordinate.

• time_varname (str) – Name of the time coordinate.

• ref_year (int) – Target year to align the file with.

• is_climatology (bool) – Whether the data is climatological.

Returns:

The number of years to add to the file’s time coordinate.

Return type:

int

pycif.plugins.datastreams.fluxes.gridded_NetCDF.write.write(self, name, flx_file, flx, mode='a', metadata=None, **kwargs)

Write a flux DataArray to a CF-style gridded NetCDF file.

Builds a dataset with time/lat/lon coordinates (plus lat_bnds/lon_bnds) from the domain in metadata, and writes it to flx_file (creating it, or appending if it already exists). If the domain is unstructured, delegates to the unstructured_NetCDF plugin’s write function instead.

Parameters:

• self – The flux tracer plugin instance.

• name (str) – Name of the flux variable to write.

• flx_file (str) – Path of the NetCDF file to write or append to.

• flx (xr.DataArray) – Flux data with time, lev, lat, lon dimensions; squeezed along lev if it has a single level.

• mode (str) – Unused directly; existence of flx_file determines whether the dataset is written (mode='w') or appended (mode='a') to the NetCDF file.

• metadata (dict, optional) – Must contain a 'domain' key with the Domain object used to derive lat/lon coordinates and bounds.

Raises:

CifKeyError – If metadata has no 'domain' key.