Excess risk analysis for epistemic uncertainty with application to variational inference

by   Futoshi Futami, et al.

We analyze the epistemic uncertainty (EU) of supervised learning in Bayesian inference by focusing on the excess risk. Existing analysis is limited to the Bayesian setting, which assumes a correct model and exact Bayesian posterior distribution. Thus we cannot apply the existing theory to modern Bayesian algorithms, such as variational inference. To address this, we present a novel EU analysis in the frequentist setting, where data is generated from an unknown distribution. We show a relation between the generalization ability and the widely used EU measurements, such as the variance and entropy of the predictive distribution. Then we show their convergence behaviors theoretically. Finally, we propose new variational inference that directly controls the prediction and EU evaluation performances based on the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing methods.


page 1

page 2

page 3

page 4


Loss function based second-order Jensen inequality and its application to particle variational inference

Bayesian model averaging, obtained as the expectation of a likelihood fu...

A Generalization Bound for Online Variational Inference

Bayesian inference provides an attractive online-learning framework to a...

Variational Inference and Bayesian CNNs for Uncertainty Estimation in Multi-Factorial Bone Age Prediction

Additionally to the extensive use in clinical medicine, biological age (...

VarFA: A Variational Factor Analysis Framework For Efficient Bayesian Learning Analytics

We propose VarFA, a variational inference factor analysis framework that...

Implicit Variational Inference: the Parameter and the Predictor Space

Having access to accurate confidence levels along with the predictions a...

Fully Bayesian VIB-DeepSSM

Statistical shape modeling (SSM) enables population-based quantitative a...

Stochastic Neural Network with Kronecker Flow

Recent advances in variational inference enable the modelling of highly ...

Please sign up or login with your details

Forgot password? Click here to reset