DeepAI AI Chat
Log In Sign Up

Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

by   Tom Huix, et al.

This paper studies the Variational Inference (VI) used for training Bayesian Neural Networks (BNN) in the overparameterized regime, i.e., when the number of neurons tends to infinity. More specifically, we consider overparameterized two-layer BNN and point out a critical issue in the mean-field VI training. This problem arises from the decomposition of the lower bound on the evidence (ELBO) into two terms: one corresponding to the likelihood function of the model and the second to the Kullback-Leibler (KL) divergence between the prior distribution and the variational posterior. In particular, we show both theoretically and empirically that there is a trade-off between these two terms in the overparameterized regime only when the KL is appropriately re-scaled with respect to the ratio between the the number of observations and neurons. We also illustrate our theoretical results with numerical experiments that highlight the critical choice of this ratio.


page 1

page 2

page 3

page 4


Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

Variational inference enables approximate posterior inference of the hig...

Forward Amortized Inference for Likelihood-Free Variational Marginalization

In this paper, we introduce a new form of amortized variational inferenc...

Alpha-Divergences in Variational Dropout

We investigate the use of alternative divergences to Kullback-Leibler (K...

On the Performance of Direct Loss Minimization for Bayesian Neural Networks

Direct Loss Minimization (DLM) has been proposed as a pseudo-Bayesian me...

Wide Mean-Field Bayesian Neural Networks Ignore the Data

Bayesian neural networks (BNNs) combine the expressive power of deep lea...

Gaussian Mean Field Regularizes by Limiting Learned Information

Variational inference with a factorized Gaussian posterior estimate is a...

Bayesian Neural Networks With Maximum Mean Discrepancy Regularization

Bayesian Neural Networks (BNNs) are trained to optimize an entire distri...