Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping

by   Jia-Qi Yang, et al.

Research has shown that deep networks tend to be overly optimistic about their predictions, leading to an underestimation of prediction errors. Due to the limited nature of data, existing studies have proposed various methods based on model prediction probabilities to bin the data and evaluate calibration error. We propose a more generalized definition of calibration error called Partitioned Calibration Error (PCE), revealing that the key difference among these calibration error metrics lies in how the data space is partitioned. We put forth an intuitive proposition that an accurate model should be calibrated across any partition, suggesting that the input space partitioning can extend beyond just the partitioning of prediction probabilities, and include partitions directly related to the input. Through semantic-related partitioning functions, we demonstrate that the relationship between model accuracy and calibration lies in the granularity of the partitioning function. This highlights the importance of partitioning criteria for training a calibrated and accurate model. To validate the aforementioned analysis, we propose a method that involves jointly learning a semantic aware grouping function based on deep model features and logits to partition the data space into subsets. Subsequently, a separate calibration function is learned for each subset. Experimental results demonstrate that our approach achieves significant performance improvements across multiple datasets and network architectures, thus highlighting the importance of the partitioning function for calibration.


page 2

page 7


Beyond calibration: estimating the grouping loss of modern neural networks

Good decision making requires machine-learning models to provide trustwo...

A Consistent and Differentiable Lp Canonical Calibration Error Estimator

Calibrated probabilistic classifiers are models whose predicted probabil...

Learning and Testing Variable Partitions

Let F be a multivariate function from a product set Σ^n to an Abelian gr...

Spline-Based Probability Calibration

In many classification problems it is desirable to output well-calibrate...

Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics

Abstract Post hoc recalibration of prediction uncertainties of machine l...

Calibration of Neural Networks using Splines

Calibrating neural networks is of utmost importance when employing them ...

AutoAtlas: Neural Network for 3D Unsupervised Partitioning and Representation Learning

We present a novel neural network architecture called AutoAtlas for full...

Please sign up or login with your details

Forgot password? Click here to reset