DeepAI AI Chat
Log In Sign Up

On the Difficulty of Epistemic Uncertainty Quantification in Machine Learning: The Case of Direct Uncertainty Estimation through Loss Minimisation

by   Viktor Bengs, et al.
Ghent University
Universität München

Uncertainty quantification has received increasing attention in machine learning in the recent past. In particular, a distinction between aleatoric and epistemic uncertainty has been found useful in this regard. The latter refers to the learner's (lack of) knowledge and appears to be especially difficult to measure and quantify. In this paper, we analyse a recent proposal based on the idea of a second-order learner, which yields predictions in the form of distributions over probability distributions. While standard (first-order) learners can be trained to predict accurate probabilities, namely by minimising suitable loss functions on sample data, we show that loss minimisation does not work for second-order predictors: The loss functions proposed for inducing such predictors do not incentivise the learner to represent its epistemic uncertainty in a faithful way.


page 1

page 2

page 3

page 4


On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

It is well known that accurate probabilistic predictors can be trained t...

On Calibration of Ensemble-Based Credal Predictors

In recent years, several classification methods that intend to quantify ...

Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference

The idea to distinguish and quantify two important types of uncertainty,...

Aleatoric and Epistemic Uncertainty with Random Forests

Due to the steadily increasing relevance of machine learning for practic...

Epistemic Deep Learning

The belief function approach to uncertainty quantification as proposed i...

Evaluating subgroup disparity using epistemic uncertainty in mammography

As machine learning (ML) continue to be integrated into healthcare syste...

Adaptive reconstruction of imperfectly-observed monotone functions, with applications to uncertainty quantification

Motivated by the desire to numerically calculate rigorous upper and lowe...