pycif.plugins.modes.ensrf — API reference

pycif.plugins.modes.ensrf — API reference#

Configuration reference: ensrf plugin

pycif.plugins.modes.ensrf.build_H.build_Hx(obsvect, segment_dir, nsample, ddi, ddf)[source]#

Builds the observation operator matrix based on the simulated values of each member.

Parameters:

  • obsvect (Plugin) – obsvect Plugin

  • segment_dir (str) – path to the segment directory

  • nsample (int) – number of members in the ensemble (besides prior and mean members)

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

Returns:

matrix representing the observation operator for each member

Return type:

harray (np.array)

pycif.plugins.modes.ensrf.execute.execute(self, **kwargs)[source]#

Performs an EnSRF inversion based on Peters et al. (2005).

Parameters:

self (Plugin) – mode Plugin

Returns:

optimized controlvect Plugin obsvect: optimized obsvect Plugin

Return type:

controlvect

pycif.plugins.modes.ensrf.localization.dist_obs_matrix(zlon1, zlat1, zlon2, zlat2, projection='gps')[source]#

Computes the distance vector between (zlon1, zlat1) and an observation’s location.

Parameters:

  • zlat1 (np.array) – numpy array of latitudes

  • zlon1 (np.array) – numpy array of longitudes

  • projection (str) – the projection used for the longitudes and latitudes

  • lon_obs (float) – the projection used for the longitudes and latitudes

  • lat_obs (float) – the projection used for the longitudes and latitudes

Returns:

vector of distances

Return type:

np.array(zlat1.size, zlat2.size)

pycif.plugins.modes.ensrf.localization.fill_loc_matrices(mode, controlvect, lons_obs2assim, lats_obs2assim, list_cntrlv_idx)[source]#

Fill the localization matrices based on the requested decay function and the distance between pixels and observations.

Parameters:

  • mode (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • lons_obs (np.array) – longitudes of observations

  • lats_obs (np.array) – latitudes of observations

Returns:

filled localization matrix

Return type:

localize matrix (np.array)

pycif.plugins.modes.ensrf.localization.apply_function_loc(dx, decay_func, decay_length)[source]#

Apply the decay function to generate the localization.

Parameters:

  • dx – the distance matrix

  • decay_func – the decay function

  • decay_length – the decay length

Returns:

obsvect updated with latitude, longitude, date, enddate and parameter

pycif.plugins.modes.ensrf.localization.complement_obsvect(obsvect, **kwargs)[source]#

Add extra information to obsvect needed for EnSRF.

Parameters:

obsvect (Plugin) – the observation vector with all its attributes

Returns:

obsvect updated with latitude, longitude, date, enddate and parameter

class pycif.plugins.modes.ensrf.metrics.Metrics_ensrf(controlvect, obsvect, window_length, windows_datei, nlag)[source]#

Bases: object

update(level, segments_datei, iseg, seg_ddi, seg_ddf, controlvect, obsvect, mask_obsa, mask_obs_assimilated, x_samples, x_samples_segprior, hx_samples, sim_segprior, sim_post)[source]#
compute_obs_metrics(obsvect, iseg, wdi, is_prior_window, is_post_window, mask_obsa_w, sim_prior, sim_segprior, sim_post)[source]#
compute_dof_b_hresol(controlvect)[source]#

Compute the degrees of freedom of the inversion problem (B matrix) for the horizontal direction. It follows Patil et al. (2001) and Bretherton et al. (1999).

compute_dof_ensemble(iseg, wdis, x_samples_dev_prior, x_samples_dev, mask_cntrlv_idx)[source]#

Compute the degrees of freedom of the optimized ensemble following Patil et al. (2001) and Bretherton et al. (1999).

compute_analysis_metrics(obsvect, iseg, wdi, hxs_dev, mask_obsa)[source]#

Compute the influence matrix and the averaging kernel matrix following Cardinali et al. (2004) and Kim et al. (2014).

compute_cost_function(obsvect, iseg, wdi, is_post_window, chi_prior_seg, chi_post, sim_prior, sim_segprior, sim_post, mask_obsa_w, mask_cntrlv_idx_w)[source]#

Compute individual cost function for each window.

final_update(obsvect, controlvect, sim_prior, sim_post, hx_samples)[source]#

Compute total cost function, RMSE and RMSD for the full period.

dump(filename)[source]#
pycif.plugins.modes.ensrf.optimization.serial_optimization(mode, controlvect, nsample, list_obs2assim_idx, list_cntrlv_idx, yobs, yobs_err, x_samples_dev, hx_mean, hx_samples_dev, loc_matrix_state, loc_matrix_obs)[source]#

Serial optimization following Whitacker and Hamill (2002)

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • list_obs2assim_idx (list)

  • list_cntrlv_idx (list)

  • yobs

  • yobs_err

  • x_samples_dev

  • hx_mean

  • hx_samples_dev

  • loc_matrix_state

  • loc_matrix_obs

Returns:

x_samples_dev: hx_mean: hx_samples_dev:

Return type:

xa

pycif.plugins.modes.ensrf.optimization.bulk_optimization(mode, controlvect, nsample, list_obs2assim_idx, list_cntrlv_idx, yobs, yobs_err, x_samples_dev, hx_mean, hx_samples_dev, loc_matrix_state, loc_matrix_obs)[source]#

Bulk optimization following Whitacker and Hamill (2002). See also this other version of the paper for a better explanation. https://ams.confex.com/ams/pdfpapers/23823.pdf

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • list_obs2assim_idx (list)

  • list_cntrlv_idx (list)

  • yobs

  • yobs_err

  • x_samples_dev

  • hx_mean

  • hx_samples_dev

  • loc_matrix_state

Returns:

x_samples_dev: hx_mean: hx_samples_dev:

Return type:

xa

pycif.plugins.modes.ensrf.sampling.draw_normal_samples(self, controlvect, nsample, set_dev_equal=True)[source]#

Draw a new ensemble of nsample members for the full assimilation window based on the prior mean (xb), the B matrix and a normal distribution.

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • nsample (int) – number of members (besides prior and mean)

Returns:

created members

Return type:

x_samples (np.array)

pycif.plugins.modes.ensrf.sampling.generate_ensemble_full_window(self, controlvect, nsample, set_dev_equal=True)[source]#

Draw a new ensemble of nsample + 3 members for the full assimilation window based on the prior mean (xb), the B matrix and a normal distribution.

First member is only 1.0 (for propagating unperturbed flux) Second member is the prior (for calculating the prior simulation) Third member is the mean / analysis (for calculating the posterior simulation)

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • nsample (int) – number of members (besides prior and mean)

Returns:

created members

Return type:

x_samples (np.array)

pycif.plugins.modes.ensrf.sampling.update_ensemble_new_segment(self, controlvect, x_samples, ddi, ddf, list_cntrlv_idx, set_dev_equal=False)[source]#

When a new segment/cycle is started, this function updates the mean of the new window based on the posterior mean of the window before it. The new mean is a linear combination of the prior mean and the posterior mean of the previous window. This step can be seen as a very simple forecast step.

A new ensemble is then drawn based on this new mean and the existing ensemble for this new window is filled with these new members.

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • x_samples (np.array) – existing members

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

  • set_dev_equal – Force the deviations of scaling factors for different windows in the same segment to be equal. The aim is to limit spurious anticorrelations between the same control variables in two different windows that could limit the analysis.

Returns:

updated members

Return type:

x_samples (np.array)

pycif.plugins.modes.ensrf.sampling.run_ensemble(self, controlvect, nsample, x_samples, segment_dir, ddi, ddf, same_restart=True)[source]#

Dump the controlvect and run the ensemble with a batch-sampling method or a separate-sampling method.

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • nsample (int) – number of members (besides prior and mean)

  • x_samples (np.array) – existing members

  • segment_dir (str) – path to the segment directory

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

  • same_restart (boolean) – whether samples in restart_inicond should be set equal to the mean or not

Returns:

None

pycif.plugins.modes.ensrf.sampling.batch_sampling(self, controlvect, segment_dir, nsample, ddi, ddf, same_restart=True)[source]#
Create new Yaml configuration files using the batch_computation option

and launch a job to perform the batch sampling.

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • segment_dir (str) – path to the segment directory

  • nsample (int) – number of members (besides prior and mean)

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

  • same_restart (boolean) – whether samples in restart_inicond should be set equal to the mean or not

Returns:

None

pycif.plugins.modes.ensrf.sampling.separate_sampling(self, controlvect, segment_dir, nsample, sample_chunks, ddi, ddf, same_restart=True)[source]#

Create new Yaml configuration files for each member and launch multiple jobs to perform each sampling.

Parameters:

  • self (Plugin) – mode Plugin

  • controlvect (Plugin) – controlvect Plugin

  • segment_dir (str) – path to the segment directory

  • nsample (int) – number of members (besides prior and mean)

  • sample_chunks (list) – list of sample indices in each chunk

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

  • same_restart (boolean) – whether samples in restart_inicond should be set equal to the mean or not

Returns:

None

pycif.plugins.modes.ensrf.segment_cropping.select_obs2assimilate(obsvect, hx_samples_dev, mask_obs_assimilated)[source]#

Apply multiple operations to select the information (observations, controlvect) that must be considered/updated as part of the current optimization segment.

  1. Select the observations that must be assimilated

  2. Select the part of the controlvector that must be updated

Parameters:

  • obsvect (Plugin) – obsvect Plugin

  • hx_samples_dev (str) – for each member, deviations of the simulated values from the mean

  • mask_obs_assimilated (np.array) – mask indicating the observations already assimilated

Returns:

list of observation indexes that must be assimilated hx_samples_dev (str): updated deviations mask_obs_assimilated (np.array): updated mask of assimilated observations

Return type:

list_obs2assim_idx (list)

pycif.plugins.modes.ensrf.segment_cropping.get_cntrlv_trdates(controlvect)[source]#

Fetch the dates for each component and each tracer of the controlvector.

Parameters:

controlvect (Plugin) – controlvect Plugin

Returns:

Dictionary with list of dates for each component and each tracer.

Return type:

dict_components_dates (dict)

pycif.plugins.modes.ensrf.segment_cropping.get_cntrlv_idx_segment(controlvect, ddi, ddf)[source]#

Find the controlvect indexes in the current segment

Parameters:

  • controlvect (Plugin) – controlvect Plugin

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

Returns:

indexes of the controlvect that are in the segment

Return type:

list_cntrlv_idx (dict)

pycif.plugins.modes.ensrf.segment_cropping.get_cntrlv_idx_comp(controlvect, ddi, ddf)[source]#

Find the controlvect indexes for each optimized component.

Parameters:

  • controlvect (Plugin) – controlvect Plugin

  • ddi (datetime) – initial date of the segment

  • ddf (datetime) – end date of the segment

Returns:

indexes of the controlvect that are in the segment

Return type:

list_cntrlv_idx (dict)

pycif.plugins.modes.ensrf.segment_cropping.crop_dump(cntrlv, cntrlv_file, ddi_crop, ddf_crop, ensemble=False, dump_indexes=False, **kwargs)[source]#

Dumps the controlvector into a picke file. Only the elements that are inside the crop window are dumped.

Parameters:

  • cntrlv (pycif.utils.classes.controlvects.ControlVect) – the Control Vector to dump

  • cntrlv_file (str) – path to the file to dump as pickle

  • ddi_crop (datetime) – cropping start date

  • ddf_crop (datetime) – cropping end date

pycif.plugins.modes.ensrf.utils.get_path_in_dict(d, value, prepath=())[source]#

Find occurences of a value in a dictionary (or a list) recursively and returns a list of the paths leading to this occurence.

Parameters:

  • d (dict or list) – Dictionary or list in which value is searched for.

  • value (str) – the string element that is searched for in the keys or values.

  • prepath (tuple) – path to pre-append. Mainly here for recursivity.

Returns:

list of paths where the value has been found.

Return type:

list_paths (list)

pycif.plugins.modes.ensrf.utils.get_trids_dontpropagate(yml_dict, controlvect)[source]#

List including the controlvect trids that should not be affected by the propagation of the perturbed samples. They include the trids with a False ‘iscontrol’ parameter and their successors in the transforms mentioned in the Yaml.

Parameters:

controlvect (Plugin) – controlvect Plugin

Returns:

List including the controlvect trids that should not be affected by the propagation of the perturbed samples.

Return type:

list_trids_dp (list)

pycif.plugins.modes.ensrf.utils.get_trids_dontpropagate_obsvect(yml_dict, controlvect)[source]#

List including the controlvect trids that should be affected by the propagation of the perturbed samples. They include the trids with an attribute dont_propagate_obsvect: True.

Parameters:

controlvect (Plugin) – controlvect Plugin

Returns:

List including the controlvect

trids that should be affected by the propagation of the perturbed samples.

Return type:

list_trids_dpo (list)

pycif.plugins.modes.ensrf.utils.check_tresol(controlvect, window_length)[source]#

Check the tresol argument of the optimized components. It must be equal to the window_length.

Parameters:

  • controlvect (Plugin) – controlvect Plugin

  • window_length (string) – EnSRF window length