Should Ensemble Members Be Calibrated?

by   Xixin Wu, et al.

Underlying the use of statistical approaches for a wide range of applications is the assumption that the probabilities obtained from a statistical model are representative of the "true" probability that event, or outcome, will occur. Unfortunately, for modern deep neural networks this is not the case, they are often observed to be poorly calibrated. Additionally, these deep learning approaches make use of large numbers of model parameters, motivating the use of Bayesian, or ensemble approximation, approaches to handle issues with parameter estimation. This paper explores the application of calibration schemes to deep ensembles from both a theoretical perspective and empirically on a standard image classification task, CIFAR-100. The underlying theoretical requirements for calibration, and associated calibration criteria, are first described. It is shown that well calibrated ensemble members will not necessarily yield a well calibrated ensemble prediction, and if the ensemble prediction is well calibrated its performance cannot exceed that of the average performance of the calibrated ensemble members. On CIFAR-100 the impact of calibration for ensemble prediction, and associated calibration is evaluated. Additionally the situation where multiple different topologies are combined together is discussed.


page 1

page 2

page 3

page 4


Diverse Ensembles Improve Calibration

Modern deep neural networks can produce badly calibrated predictions, es...

A Note on "Assessing Generalization of SGD via Disagreement"

Jiang et al. (2021) give empirical evidence that the average test error ...

Improving robustness and calibration in ensembles with diversity regularization

Calibration and uncertainty estimation are crucial topics in high-risk e...

Better Boosting with Bandits for Online Learning

Probability estimates generated by boosting ensembles are poorly calibra...

Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy perf...

Adaptive and Calibrated Ensemble Learning with Dependent Tail-free Process

Ensemble learning is a mainstay in modern data science practice. Convent...

Please sign up or login with your details

Forgot password? Click here to reset