improver.ensemble_copula_coupling.ensemble_copula_coupling module

This module defines the plugins required for Ensemble Copula Coupling.

class improver.ensemble_copula_coupling.ensemble_copula_coupling.EnsembleReordering

Bases: improver.BasePlugin

Plugin for applying the reordering step of Ensemble Copula Coupling, in order to generate ensemble realizations with multivariate structure from percentiles. The percentiles are assumed to be in ascending order.

Reference: Schefzik, R., Thorarinsdottir, T.L. & Gneiting, T., 2013. Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling. Statistical Science, 28(4), pp.616-640.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

static _recycle_raw_ensemble_realizations(post_processed_forecast_percentiles, raw_forecast_realizations, percentile_coord_name)

Function to determine whether there is a mismatch between the number of percentiles and the number of raw forecast realizations. If more percentiles are requested than ensemble realizations, then the ensemble realizations are recycled. This assumes that the identity of the ensemble realizations within the raw ensemble forecast is random, such that the raw ensemble realizations are exchangeable. If fewer percentiles are requested than ensemble realizations, then only the first n ensemble realizations are used.

Parameters

• post_processed_forecast_percentiles (iris.cube.Cube) – Cube for post-processed percentiles. The percentiles are assumed to be in ascending order.

• raw_forecast_realizations (iris.cube.Cube) – Cube containing the raw (not post-processed) forecasts.

• percentile_coord_name (str) – Name of required percentile coordinate.

Returns

Cube for the raw ensemble forecast, where the raw ensemble realizations have either been recycled or constrained, depending upon the number of percentiles present in the post-processed forecast cube.

Return type

iris cube.Cube

process(post_processed_forecast, raw_forecast, random_ordering=False, random_seed=None)

Reorder post-processed forecast using the ordering of the raw ensemble.

Parameters

• post_processed_forecast (iris.cube.Cube) – The cube containing the post-processed forecast realizations.

• raw_forecast (iris.cube.Cube) – The cube containing the raw (not post-processed) forecast.

• random_ordering (bool) – If random_ordering is True, the post-processed forecasts are reordered randomly, rather than using the ordering of the raw ensemble.

• random_seed (int) – If random_seed is an integer, the integer value is used for the random seed. If random_seed is None, no random seed is set, so the random values generated are not reproducible.

Returns

Cube containing the new ensemble realizations where all points within the dataset have been reordered in comparison to the input percentiles.

Return type

iris.cube.Cube

static rank_ecc(post_processed_forecast_percentiles, raw_forecast_realizations, random_ordering=False, random_seed=None)

Function to apply Ensemble Copula Coupling. This ranks the post-processed forecast realizations based on a ranking determined from the raw forecast realizations.

Parameters

• post_processed_forecast_percentiles (iris.cube.Cube) – Cube for post-processed percentiles. The percentiles are assumed to be in ascending order.

• raw_forecast_realizations (iris.cube.Cube) – Cube containing the raw (not post-processed) forecasts. The probabilistic dimension is assumed to be the zeroth dimension.

• random_ordering (bool) – If random_ordering is True, the post-processed forecasts are reordered randomly, rather than using the ordering of the raw ensemble.

• random_seed (int or None) – If random_seed is an integer, the integer value is used for the random seed. If random_seed is None, no random seed is set, so the random values generated are not reproducible.

Returns

Cube for post-processed realizations where at a particular grid point, the ranking of the values within the ensemble matches the ranking from the raw ensemble.

Return type

iris.cube.Cube

class improver.ensemble_copula_coupling.ensemble_copula_coupling.FromMeanAndVariance(distribution='norm', shape_parameters=None)

Bases: object

Base Class to support the plugins that compute percentiles and probabilities from the mean and variance.

__init__(distribution='norm', shape_parameters=None)

Initialise the class.

Parameters

• distribution (str) – Name of a distribution supported by scipy.stats.

• shape_parameters (list or None) – For use with distributions in scipy.stats (e.g. truncnorm) that require the specification of shape parameters to be able to define the shape of the distribution. For the truncated normal distribution, the shape parameters should be appropriate for distribution constructed from the mean and standard deviation provided. Please note that for use with calculate_truncated_normal_crps(), the shape parameters for a truncated normal distribution with a lower bound of zero should be [0, np.inf].

_rescale_shape_parameters(mean, std)

Rescale the shape parameters for the desired mean and standard deviation for the truncated normal distribution. The shape parameters for any other distribution will remain unchanged.

For the truncated normal distribution, if the shape parameters are not rescaled, then scipy.stats.truncnorm will assume that the shape parameters are appropriate for a standard normal distribution. As the aim is to construct a distribution using specific values for the mean and standard deviation, the assumption of a standard normal distribution is not appropriate. Therefore the shape parameters are rescaled using the equations:

\[ \begin{align}\begin{aligned}a\_rescaled = (a - mean)/standard\_deviation\\b\_rescaled = (b - mean)/standard\_deviation\end{aligned}\end{align} \]

Please see scipy.stats.truncnorm for some further information.

Parameters

• mean (numpy.ndarray) – Mean to be used to scale the shape parameters.

• std (numpy.ndarray) – Standard deviation to be used to scale the shape parameters.

class improver.ensemble_copula_coupling.ensemble_copula_coupling.GeneratePercentilesFromMeanAndVariance(distribution='norm', shape_parameters=None)

Bases: improver.BasePlugin, improver.ensemble_copula_coupling.ensemble_copula_coupling.FromMeanAndVariance

Plugin focussing on generating percentiles from mean and variance. In combination with the EnsembleReordering plugin, this is Ensemble Copula Coupling.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

_mean_and_variance_to_percentiles(calibrated_forecast_predictor, calibrated_forecast_variance, template_cube, percentiles)

Function returning percentiles based on the supplied mean and variance. The percentiles are created by assuming a Gaussian distribution and calculating the value of the phenomenon at specific points within the distribution.

Parameters

• calibrated_forecast_predictor (iris.cube.Cube) – Predictor for the calibrated forecast i.e. the mean.

• calibrated_forecast_variance (iris.cube.Cube) – Variance for the calibrated forecast.

• template_cube (iris.cube.Cube) – Template cube containing either a percentile or realization coordinate. All coordinates apart from the percentile or realization coordinate will be copied from the template cube. Metadata will also be copied from this cube.

• percentiles (list) – Percentiles at which to calculate the value of the phenomenon at.

Returns

Cube containing the values for the phenomenon at each of the percentiles requested.

Return type

iris.cube.Cube

Raises

ValueError – If any of the resulting percentile values are nans and these nans are not caused by a zero variance.

process(calibrated_forecast_predictor, calibrated_forecast_variance, template_cube, no_of_percentiles=None, percentiles=None)

Generate ensemble percentiles from the mean and variance.

Parameters

• calibrated_forecast_predictor (iris.cube.Cube) – Cube containing the calibrated forecast predictor.

• calibrated_forecast_variance (iris.cube.Cube) – Cube containing the calibrated forecast variance.

• template_cube (iris.cube.Cube) – Template cube containing either a percentile or realization coordinate. All coordinates apart from the percentile or realization coordinate will be copied from the template cube. Metadata will also be copied from this cube.

• no_of_percentiles (int) – Integer defining the number of percentiles that will be calculated from the mean and variance.

• percentiles (list) – List of percentiles that will be generated from the mean and variance provided.

Returns

Cube for calibrated percentiles. The percentile coordinate is always the zeroth dimension.

Return type

iris.cube.Cube

Raises

ValueError – Ensure that it is not possible to supply “no_of_percentiles” and “percentiles” simultaneously as keyword arguments.

class improver.ensemble_copula_coupling.ensemble_copula_coupling.GeneratePercentilesFromProbabilities(ecc_bounds_warning=False)

Bases: improver.BasePlugin

Class for generating percentiles from probabilities. In combination with the Ensemble Reordering plugin, this is a variant Ensemble Copula Coupling.

This class includes the ability to interpolate between probabilities specified using multiple thresholds in order to generate the percentiles, see Figure 1 from Flowerdew, 2014.

Scientific Reference: Flowerdew, J., 2014. Calibrated ensemble reliability whilst preserving spatial structure. Tellus Series A, Dynamic Meteorology and Oceanography, 66, 22662.

__init__(ecc_bounds_warning=False)

Initialise the class.

Parameters

ecc_bounds_warning (bool) – If true and ECC bounds are exceeded by the percentile values, a warning will be generated rather than an exception. Default value is FALSE.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

_add_bounds_to_thresholds_and_probabilities(threshold_points, probabilities_for_cdf, bounds_pairing)

Padding of the lower and upper bounds of the distribution for a given phenomenon for the threshold_points, and padding of probabilities of 0 and 1 to the forecast probabilities.

Parameters

• threshold_points (numpy.ndarray) – Array of threshold values used to calculate the probabilities.

• probabilities_for_cdf (numpy.ndarray) – Array containing the probabilities used for constructing an cumulative distribution function i.e. probabilities below threshold.

• bounds_pairing (tuple) – Lower and upper bound to be used as the ends of the cumulative distribution function.

Returns

tuple containing:

threshold_points (numpy.ndarray):

Array of threshold values padded with the lower and upper bound of the distribution.

probabilities_for_cdf (numpy.ndarray):

Array containing the probabilities padded with 0 and 1 at each end.

Return type

(tuple)

Raises

ValueError – If the thresholds exceed the ECC bounds for the diagnostic and self.ecc_bounds_warning is False.

Warns

Warning – If the thresholds exceed the ECC bounds for the diagnostic and self.ecc_bounds_warning is True.

_probabilities_to_percentiles(forecast_probabilities, percentiles, bounds_pairing)

Conversion of probabilities to percentiles through the construction of an cumulative distribution function. This is effectively constructed by linear interpolation from the probabilities associated with each threshold to a set of percentiles.

Parameters

• forecast_probabilities (iris.cube.Cube) – Cube with a threshold coordinate.

• percentiles (numpy.ndarray) – Array of percentiles, at which the corresponding values will be calculated.

• bounds_pairing (tuple) – Lower and upper bound to be used as the ends of the cumulative distribution function.

Returns

Cube containing values for the required diagnostic e.g. air_temperature at the required percentiles.

Return type

iris.cube.Cube

Raises

NotImplementedError – If the threshold coordinate has an spp__relative_to_threshold attribute that is not either “above” or “below”.

Warns

Warning – If the probability values are not ascending, so the resulting cdf is not monotonically increasing.

process(forecast_probabilities, no_of_percentiles=None, percentiles=None, sampling='quantile')

1. Concatenates cubes with a threshold coordinate.

2. Creates a list of percentiles.

3. Accesses the lower and upper bound pair to find the ends of the cumulative distribution function.

4. Convert the threshold coordinate into values at a set of percentiles using linear interpolation, see Figure 1 from Flowerdew, 2014.

Parameters

• forecast_probabilities (iris.cube.Cube) – Cube containing a threshold coordinate.

• no_of_percentiles (int) – Number of percentiles. If None and percentiles is not set, the number of thresholds within the input forecast_probabilities cube is used as the number of percentiles. This argument is mutually exclusive with percentiles.

• percentiles (list of float) – The desired percentile values in the interval [0, 100]. This argument is mutually exclusive with no_of_percentiles.

• sampling (str) –

  Type of sampling of the distribution to produce a set of percentiles e.g. quantile or random.

  Accepted options for sampling are:

  • Quantile: A regular set of equally-spaced percentiles aimed

    at dividing a Cumulative Distribution Function into blocks of equal probability.

  • Random: A random set of ordered percentiles.

Returns

Cube with forecast values at the desired set of percentiles. The threshold coordinate is always the zeroth dimension.

Return type

iris.cube.Cube

Raises

ValueError – If both no_of_percentiles and percentiles are provided

class improver.ensemble_copula_coupling.ensemble_copula_coupling.GenerateProbabilitiesFromMeanAndVariance(distribution='norm', shape_parameters=None)

Bases: improver.BasePlugin, improver.ensemble_copula_coupling.ensemble_copula_coupling.FromMeanAndVariance

Plugin to generate probabilities relative to given thresholds from the mean and variance of a distribution.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

static _check_template_cube(cube)

The template cube is expected to contain a leading threshold dimension followed by spatial (y/x) dimensions. This check raises an error if this is not the case. If the cube contains the expected dimensions, a threshold leading order is enforced.

Parameters

cube (iris.cube.Cube) – A cube whose dimensions are checked to ensure they match what is expected.

Raises

ValueError – If cube is not of the expected dimensions.

static _check_unit_compatibility(mean_values, variance_values, probability_cube_template)

The mean, variance, and threshold values come from three different cubes. They should all be in the same units, but this is a sanity check to ensure this is the case, converting units of the means and variances if possible. This has been written specifically for this plugin as we are comparing squared units in the case of the variance.

Parameters

• mean_values (iris.cube.Cube) – Cube of mean values.

• variance_values (iris.cube.Cube) – Cube of variance values.

• probability_cube_template (iris.cube.Cube) – Cube containing threshold values.

Raises

ValueError – If units of input cubes are not compatible.

_mean_and_variance_to_probabilities(mean_values, variance_values, probability_cube_template)

Function returning probabilities relative to provided thresholds based on the supplied mean and variance. A Gaussian distribution is assumed.

Parameters

• mean_values (iris.cube.Cube) – Predictor for the calibrated forecast i.e. the mean.

• variance_values (iris.cube.Cube) – Variance for the calibrated forecast.

• probability_cube_template (iris.cube.Cube) – A probability cube that has a threshold coordinate, where the probabilities are defined as above or below the threshold by the spp__relative_to_threshold attribute. This cube matches the desired output cube format.

Returns

Cube containing the data expressed as probabilities relative to the provided thresholds in the way described by spp__relative_to_threshold.

Return type

iris.cube.Cube

process(mean_values, variance_values, probability_cube_template)

Generate probabilities from the mean and variance of distribution.

Parameters

• mean_values (iris.cube.Cube) – Cube containing the distribution mean values of a diagnostic, e.g. the mean over realizations.

• variance_values (iris.cube.Cube) – Cube containing the distribution variance values of a diagnostic, e.g. the variance across realizations.

• probability_cube_template (iris.cube.Cube) – A probability cube that has a threshold coordinate, where the probabilities are defined as above or below the threshold by the spp__relative_to_threshold attribute. This cube matches the desired output cube format.

Returns

A cube of diagnostic data expressed as probabilities relative to the thresholds found in the probability_cube_template.

Return type

iris.cube.Cube

class improver.ensemble_copula_coupling.ensemble_copula_coupling.RebadgePercentilesAsRealizations

Bases: improver.BasePlugin

Class to rebadge percentiles as ensemble realizations. This will allow the quantisation to percentiles to be completed, without a subsequent EnsembleReordering step to restore spatial correlations, if required.

__init__()

Initialise the class.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

static process(cube, ensemble_realization_numbers=None)

Rebadge percentiles as ensemble realizations. The ensemble realization numbering will depend upon the number of percentiles in the input cube i.e. 0, 1, 2, 3, …, n-1, if there are n percentiles.

Parameters

• cube (iris.cube.Cube) – Cube containing a percentile coordinate, which will be rebadged as ensemble realization.

• ensemble_realization_numbers (numpy.ndarray) – An array containing the ensemble numbers required in the output realization coordinate. Default is None, meaning the realization coordinate will be numbered 0, 1, 2 … n-1 for n percentiles on the input cube.

Raises

InvalidCubeError – If the realization coordinate already exists on the cube.

class improver.ensemble_copula_coupling.ensemble_copula_coupling.ResamplePercentiles(ecc_bounds_warning=False)

Bases: improver.BasePlugin

Class for resampling percentiles from an existing set of percentiles. In combination with the Ensemble Reordering plugin, this is a variant of Ensemble Copula Coupling.

This class includes the ability to linearly interpolate from an input set of percentiles to a different output set of percentiles.

__init__(ecc_bounds_warning=False)

Initialise the class.

Parameters

ecc_bounds_warning (bool) – If true and ECC bounds are exceeded by the percentile values, a warning will be generated rather than an exception. Default value is FALSE.

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 213

_abc_registry = <_weakrefset.WeakSet object>

_add_bounds_to_percentiles_and_forecast_at_percentiles(percentiles, forecast_at_percentiles, bounds_pairing)

Padding of the lower and upper bounds of the percentiles for a given phenomenon, and padding of forecast values using the constant lower and upper bounds.

Parameters

• percentiles (numpy.ndarray) – Array of percentiles from a Cumulative Distribution Function.

• forecast_at_percentiles (numpy.ndarray) – Array containing the underlying forecast values at each percentile.

• bounds_pairing (tuple) – Lower and upper bound to be used as the ends of the cumulative distribution function.

Raises

• ValueError – If the percentile points are outside the ECC bounds and self.ecc_bounds_warning is False.

• ValueError – If the percentiles are not in ascending order.

Warns

Warning – If the percentile points are outside the ECC bounds and self.ecc_bounds_warning is True.

_interpolate_percentiles(forecast_at_percentiles, desired_percentiles, bounds_pairing, percentile_coord_name)

Interpolation of forecast for a set of percentiles from an initial set of percentiles to a new set of percentiles. This is constructed by linearly interpolating between the original set of percentiles to a new set of percentiles.

Parameters

• forecast_at_percentiles (iris.cube.Cube) – Cube containing a percentile coordinate.

• desired_percentiles (numpy.ndarray) – Array of the desired percentiles.

• bounds_pairing (tuple) – Lower and upper bound to be used as the ends of the cumulative distribution function.

• percentile_coord_name (str) – Name of required percentile coordinate.

Returns

Cube containing values for the required diagnostic e.g. air_temperature at the required percentiles.

Return type

iris.cube.Cube

process(forecast_at_percentiles, no_of_percentiles=None, sampling='quantile')

1. Creates a list of percentiles.

2. Accesses the lower and upper bound pair of the forecast values, in order to specify lower and upper bounds for the percentiles.

3. Interpolate the percentile coordinate into an alternative set of percentiles using linear interpolation.

Parameters

• forecast_at_percentiles (iris.cube.Cube) – Cube expected to contain a percentile coordinate.

• no_of_percentiles (int or None) – Number of percentiles If None, the number of percentiles within the input forecast_at_percentiles cube is used as the number of percentiles.

• sampling (str) –

  Type of sampling of the distribution to produce a set of percentiles e.g. quantile or random.

  Accepted options for sampling are:

  • Quantile: A regular set of equally-spaced percentiles aimed

    at dividing a Cumulative Distribution Function into blocks of equal probability.

  • Random: A random set of ordered percentiles.

Returns

Cube with forecast values at the desired set of percentiles. The percentile coordinate is always the zeroth dimension.

Return type

iris.cube.Cube