Evaluating forecasts for high-impact events using transformed kernel scores

by   Sam Allen, et al.

It is informative to evaluate a forecaster's ability to predict outcomes that have a large impact on the forecast user. Although weighted scoring rules have become a well-established tool to achieve this, such scores have been studied almost exclusively in the univariate case, with interest typically placed on extreme events. However, a large impact may also result from events not considered to be extreme from a statistical perspective: the interaction of several moderate events could also generate a high impact. Compound weather events provide a good example of this. To assess forecasts made for high-impact events, this work extends existing results on weighted scoring rules by introducing weighted multivariate scores. To do so, we utilise kernel scores. We demonstrate that the threshold-weighted continuous ranked probability score (twCRPS), arguably the most well-known weighted scoring rule, is a kernel score. This result leads to a convenient representation of the twCRPS when the forecast is an ensemble, and also permits a generalisation that can be employed with alternative kernels, allowing us to introduce, for example, a threshold-weighted energy score and threshold-weighted variogram score. To illustrate the additional information that these weighted multivariate scoring rules provide, results are presented for a case study in which the weighted scores are used to evaluate daily precipitation accumulation forecasts, with particular interest on events that could lead to flooding.


Weighted verification tools to evaluate univariate and multivariate forecasts for high-impact weather events

To mitigate the impacts associated with adverse weather conditions, mete...

Extreme events evaluation using CRPS distributions

Verification of ensemble forecasts for extreme events remains a challeng...

Scoring Functions for Multivariate Distributions and Level Sets

Interest in predicting multivariate probability distributions is growing...

Locally tail-scale invariant scoring rules for evaluation of extreme value forecasts

Statistical analysis of extremes can be used to predict the probability ...

A scoring framework for tiered warnings and multicategorical forecasts based on fixed risk measures

The use of tiered warnings and multicategorical forecasts are ubiquitous...

More on verification of probability forecasts for football outcomes: score decompositions, reliability, and discrimination analyses

Forecast of football outcomes in terms of Home Win, Draw and Away Win re...

On the Evaluation of Skill in Binary Forecast

A good prediction is very important for scientific, economic, and admini...

Please sign up or login with your details

Forgot password? Click here to reset