DeepAI AI Chat
Log In Sign Up

On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

by   Viktor Bengs, et al.
Ghent University
Universität München

It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.


page 1

page 2

page 3

page 4


Proper Scoring Rules for Survival Analysis

Survival analysis is the problem of estimating probability distributions...

On Calibration of Ensemble-Based Credal Predictors

In recent years, several classification methods that intend to quantify ...

Proper-Composite Loss Functions in Arbitrary Dimensions

The study of a machine learning problem is in many ways is difficult to ...

From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions

In the face of uncertainty, the need for probabilistic assessments has l...

Calibration Scoring Rules for Practical Prediction Training

In situations where forecasters are scored on the quality of their proba...

How to show a probabilistic model is better

We present a simple theoretical framework, and corresponding practical p...