Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

07/08/2022
by   Tom Huix, et al.
0

This paper studies the Variational Inference (VI) used for training Bayesian Neural Networks (BNN) in the overparameterized regime, i.e., when the number of neurons tends to infinity. More specifically, we consider overparameterized two-layer BNN and point out a critical issue in the mean-field VI training. This problem arises from the decomposition of the lower bound on the evidence (ELBO) into two terms: one corresponding to the likelihood function of the model and the second to the Kullback-Leibler (KL) divergence between the prior distribution and the variational posterior. In particular, we show both theoretically and empirically that there is a trade-off between these two terms in the overparameterized regime only when the KL is appropriately re-scaled with respect to the ratio between the the number of observations and neurons. We also illustrate our theoretical results with numerical experiments that highlight the critical choice of this ratio.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2021

Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

Variational inference enables approximate posterior inference of the hig...
research
07/10/2023

Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference

We provide a rigorous analysis of training by variational inference (VI)...
research
05/29/2018

Forward Amortized Inference for Likelihood-Free Variational Marginalization

In this paper, we introduce a new form of amortized variational inferenc...
research
11/15/2022

On the Performance of Direct Loss Minimization for Bayesian Neural Networks

Direct Loss Minimization (DLM) has been proposed as a pseudo-Bayesian me...
research
02/23/2022

Wide Mean-Field Bayesian Neural Networks Ignore the Data

Bayesian neural networks (BNNs) combine the expressive power of deep lea...
research
02/12/2019

Gaussian Mean Field Regularizes by Limiting Learned Information

Variational inference with a factorized Gaussian posterior estimate is a...
research
07/15/2023

Minimal Random Code Learning with Mean-KL Parameterization

This paper studies the qualitative behavior and robustness of two varian...

Please sign up or login with your details

Forgot password? Click here to reset