improver.ensemble_calibration.ensemble_calibration module

This module defines all the “plugins” specific for ensemble calibration.

class improver.ensemble_calibration.ensemble_calibration.ApplyCoefficientsFromEnsembleCalibration(current_forecast, optimised_coeffs, coeff_names, predictor_of_mean_flag='mean')[source]

Bases: object

Class to apply the optimised EMOS coefficients to future dates.

__init__(current_forecast, optimised_coeffs, coeff_names, predictor_of_mean_flag='mean')[source]

Create an ensemble calibration plugin that, for Nonhomogeneous Gaussian Regression, applies coefficients created using on historical forecasts and applies the coefficients to the current forecast.

Parameters:
  • current_forecast (Iris Cube or CubeList) – The Cube or CubeList containing the current forecast.
  • optimised_coeffs (Dictionary) – Dictionary containing a list of the optimised coefficients for each date.
  • coeff_names (List) – The name of each coefficient.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
_apply_params(forecast_predictors, forecast_vars, optimised_coeffs, coeff_names, predictor_of_mean_flag)[source]

Function to apply EMOS coefficients to all required dates.

Parameters:
  • forecast_predictors (Iris cube) – Cube containing the forecast predictor e.g. ensemble mean or ensemble realizations.
  • forecast_vars (Iris cube.) – Cube containing the forecast variance e.g. ensemble variance.
  • optimised_coeffs (List) – Coefficients for all dates.
  • coeff_names (List) – Coefficient names.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
Returns:

tuple containing:
calibrated_forecast_predictor_all_dates (CubeList):

List of cubes containing the calibrated forecast predictor.

calibrated_forecast_var_all_dates (CubeList):

List of cubes containing the calibrated forecast variance.

calibrated_forecast_coefficients_all_dates (CubeList):

List of cubes containing the coefficients used for calibration.
Return type:

(tuple)
_create_coefficient_cube(cube, optimised_coeffs_at_date, coeff_names)[source]

Function to create a cube to store the coefficients used in the ensemble calibration.

Parameters:
  • cube (Iterable of coordinates) – The coordinates to be checked for length one coordinates.
  • optimised_coeffs_at_date (List of coefficients) – Optimised coefficients for a particular date.
  • coeff_names (List) – List of coefficient names.
Returns:

Cube containing the coefficient value as the data array.
Return type:

coeff_cubes (Iris cube)
_find_coords_of_length_one(cube, add_dimension=True)[source]

Function to find all coordinates with a length of 1.

Parameters:
  • cube (Iris cube) – Cube
  • add_dimension (Logical) – Adds a dimension of 0 to each coordinate. A tuple is appended.
Returns:

List of length one coordinates or list of tuples containing length one coordinates and the dimension.
Return type:

length_one_coords (List or List of tuples)
_separate_length_one_coords_into_aux_and_dim(length_one_coords, dim_coords=['time'])[source]

Function to separate coordinates into auxiliary and dimension coordinates.

Parameters:
  • length_one_coords (Iterable of coordinates) – The coordinates to be checked for length one coordinates.
  • dim_coords (List of coordinates) – The length one coordinates to be made dimension coordinates.
Returns:

tuple containing:
length_one_coords_for_aux_coords (List):

List of length one coordinates to be auxiliary coordinates, i.e. not in the dim_coords list. length_one_coords_for_dim_coords (List): List of length one coordinates to be dimension coordinates, according to dim_coords list.
Return type:

(tuple)
apply_params_entry()[source]

Wrapping function to calculate the forecast predictor and forecast variance prior to applying coefficients to the current forecast.

Returns:
tuple containing:
calibrated_forecast_predictor (CubeList):
CubeList containing both the calibrated version of the ensemble predictor, either the ensemble mean/realizations.
calibrated_forecast_variance (CubeList):
CubeList containing both the calibrated version of the ensemble variance, either the ensemble mean/realizations.
calibrated_forecast_coefficients (CubeList):
CubeList containing both the coefficients for calibrating the ensemble.
Return type:(tuple)
class improver.ensemble_calibration.ensemble_calibration.ContinuousRankedProbabilityScoreMinimisers[source]

Bases: object

Minimise the Continuous Ranked Probability Score (CRPS)

Calculate the optimised coefficients for minimising the CRPS based on assuming a particular probability distribution for the phenomenon being minimised.

The number of coefficients that will be optimised depend upon the initial guess.

Minimisation is performed using the Nelder-Mead algorithm for 200 iterations to limit the computational expense. Note that the BFGS algorithm was initially trialled but had a bug in comparison to comparative results generated in R.

BAD_VALUE = 999999.0
MAX_ITERATIONS = 200
TOLERATED_PERCENTAGE_CHANGE = 5
crps_minimiser_wrapper(initial_guess, forecast_predictor, truth, forecast_var, predictor_of_mean_flag, distribution)[source]

Function to pass a given minimisation function to the scipy minimize function to estimate optimised values for the coefficients.

Parameters:
  • initial_guess (List) – List of optimised coefficients. Order of coefficients is [c, d, a, b].
  • forecast_predictor (iris.cube.Cube) – Cube containing the fields to be used as the predictor, either the ensemble mean or the ensemble realizations.
  • truth (iris.cube.Cube) – Cube containing the field, which will be used as truth.
  • forecast_var (iris.cube.Cube) – Cube containg the field containing the ensemble variance.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
  • distribution (String) – String used to access the appropriate minimisation function within self.minimisation_dict.
Returns:

List of optimised coefficients. Order of coefficients is [c, d, a, b].
Return type:

optimised_coeffs (List)
normal_crps_minimiser(initial_guess, forecast_predictor, truth, forecast_var, sqrt_pi, predictor_of_mean_flag)[source]

Minimisation function to calculate coefficients based on minimising the CRPS for a normal distribution.

Scientific Reference: Gneiting, T. et al., 2005. Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation. Monthly Weather Review, 133(5), pp.1098-1118.

Parameters:
  • initial_guess – List List of optimised coefficients. Order of coefficients is [c, d, a, b].
  • forecast_predictor – Numpy array Data to be used as the predictor, either the ensemble mean or the ensemble realizations.
  • truth – Numpy array Data to be used as truth.
  • forecast_var – Numpy array Ensemble variance data.
  • sqrt_pi – Numpy array Square root of Pi
  • predictor_of_mean_flag – String String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
Returns:

Minimum value for the CRPS achieved.
Return type:

result (Float)
truncated_normal_crps_minimiser(initial_guess, forecast_predictor, truth, forecast_var, sqrt_pi, predictor_of_mean_flag)[source]

Minimisation function to calculate coefficients based on minimising the CRPS for a truncated_normal distribution.

Scientific Reference: Thorarinsdottir, T.L. & Gneiting, T., 2010. Probabilistic forecasts of wind speed: Ensemble model output statistics by using heteroscedastic censored regression. Journal of the Royal Statistical Society. Series A: Statistics in Society, 173(2), pp.371-388.

Parameters:
  • initial_guess (List) – List of optimised coefficients. Order of coefficients is [c, d, a, b].
  • forecast_predictor (Numpy array) – Data to be used as the predictor, either the ensemble mean or the ensemble realizations.
  • truth (Numpy array) – Data to be used as truth.
  • forecast_var (Numpy array) – Ensemble variance data.
  • sqrt_pi (Numpy array) – Square root of Pi
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
Returns:

Minimum value for the CRPS achieved.
Return type:

result (Float)
class improver.ensemble_calibration.ensemble_calibration.EnsembleCalibration(calibration_method, distribution, desired_units, predictor_of_mean_flag='mean')[source]

Bases: object

Plugin to wrap the core EMOS processes: 1. Estimate optimised EMOS coefficients from training period. 2. Apply optimised EMOS coefficients for future dates.

__init__(calibration_method, distribution, desired_units, predictor_of_mean_flag='mean')[source]

Create an ensemble calibration plugin that, for Nonhomogeneous Gaussian Regression, calculates coefficients based on historical forecasts and applies the coefficients to the current forecast.

Parameters:
  • calibration_method (String) –

    The calibration method that will be applied. Supported methods are:

    ensemble model output statistics nonhomogeneous gaussian regression

    Currently these methods are not supported:

    logistic regression bayesian model averaging
  • distribution (String) – The distribution that will be used for calibration. This will be dependent upon the input phenomenon. This has to be supported by the minimisation functions in ContinuousRankedProbabilityScoreMinimisers.
  • desired_units (String or cf_units.Unit) – The unit that you would like the calibration to be undertaken in. The current forecast, historical forecast and truth will be converted as required.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
process(current_forecast, historic_forecast, truth)[source]

Performs ensemble calibration through the following steps: 1. Estimate optimised coefficients from training period. 2. Apply optimised coefficients to current forecast.

Parameters:
  • current_forecast (Iris Cube or CubeList) – The Cube or CubeList that provides the input forecast for the current cycle.
  • historic_forecast (Iris Cube or CubeList) – The Cube or CubeList that provides the input historic forecasts for calibration.
  • truth (Iris Cube or CubeList) – The Cube or CubeList that provides the input truth for calibration with dates matching the historic forecasts.
Returns:

CubeList containing the calibrated forecast predictor and calibrated forecast variance.
Return type:

calibrated_forecast_predictor_and_variance (CubeList)
class improver.ensemble_calibration.ensemble_calibration.EstimateCoefficientsForEnsembleCalibration(distribution, desired_units, predictor_of_mean_flag='mean')[source]

Bases: object

Class focussing on estimating the optimised coefficients for ensemble calibration.

ESTIMATE_COEFFICIENTS_FROM_LINEAR_MODEL_FLAG = True
__init__(distribution, desired_units, predictor_of_mean_flag='mean')[source]

Create an ensemble calibration plugin that, for Nonhomogeneous Gaussian Regression, calculates coefficients based on historical forecasts and applies the coefficients to the current forecast.

Parameters:
  • distribution (String) – Name of distribution. Assume that the current forecast can be represented using this distribution.
  • desired_units (String or cf_units.Unit) – The unit that you would like the calibration to be undertaken in. The current forecast, historical forecast and truth will be converted as required.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
compute_initial_guess(truth, forecast_predictor, predictor_of_mean_flag, estimate_coefficients_from_linear_model_flag, no_of_realizations=None)[source]

Function to compute initial guess of the a and beta components of the EMOS coefficients by linear regression of the forecast predictor and the truth, if requested. Otherwise, default values for a and b will be used.

Default values have been chosen based on Figure 8 in the 2017 ensemble calibration report available on the Science Plugin Documents Confluence page.

Parameters:
  • truth (Iris cube) – Cube containing the field, which will be used as truth.
  • forecast_predictor (Iris cube) – Cube containing the fields to be used as the predictor, either the ensemble mean or the ensemble realizations.
  • predictor_of_mean_flag (String) – String to specify the input to calculate the calibrated mean. Currently the ensemble mean (“mean”) and the ensemble realizations (“realizations”) are supported as the predictors.
  • estimate_coefficients_from_linear_model_flag (Logical) – Flag whether coefficients should be estimated from the linear regression, or static estimates should be used.
  • no_of_realizations (Int) – Number of realizations, if ensemble realizations are to be used as predictors. Default is None.
Returns:

List of coefficients to be used as initial guess. Order of coefficients is [c, d, a, b].
Return type:

initial_guess (List)
estimate_coefficients_for_ngr(current_forecast, historic_forecast, truth)[source]

Using Nonhomogeneous Gaussian Regression/Ensemble Model Output Statistics, estimate the required coefficients from historical forecasts.

The main contents of this method is:

  1. Metadata checks to ensure that the current forecast, historic forecast and truth exist in a form that can be processed.
  2. Loop through times within the concatenated current forecast cube:
    1. Extract the desired forecast period from the historic forecasts to match the current forecasts. Apply unit conversion to ensure that historic forecasts have the desired units for calibration.
    2. Extract the relevant truth to co-incide with the time within the historic forecasts. Apply unit conversion to ensure that the truth has the desired units for calibration.
    3. Calculate mean and variance.
    4. Calculate initial guess at coefficient values by performing a linear regression, if requested, otherwise default values are used.
    5. Perform minimisation.
Parameters:
  • current_forecast (Iris Cube or CubeList) – The cube containing the current forecast.
  • historical_forecast (Iris Cube or CubeList) – The cube or cubelist containing the historical forecasts used for calibration.
  • truth (Iris Cube or CubeList) – The cube or cubelist containing the truth used for calibration.
Returns:

tuple containing:
optimised_coeffs (Dictionary):

Dictionary containing a list of the optimised coefficients for each date.

coeff_names (List):

The name of each coefficient.
Return type:

(tuple)