Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

06/13/2021
by   Beau Coker, et al.
0

Variational inference enables approximate posterior inference of the highly over-parameterized neural networks that are popular in modern machine learning. Unfortunately, such posteriors are known to exhibit various pathological behaviors. We prove that as the number of hidden units in a single-layer Bayesian neural network tends to infinity, the function-space posterior mean under mean-field variational inference actually converges to zero, completely ignoring the data. This is in contrast to the true posterior, which converges to a Gaussian process. Our work provides insight into the over-regularization of the KL divergence in variational inference.

READ FULL TEXT

page 13

page 14

research
02/23/2022

Wide Mean-Field Bayesian Neural Networks Ignore the Data

Bayesian neural networks (BNNs) combine the expressive power of deep lea...
research
07/08/2022

Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

This paper studies the Variational Inference (VI) used for training Baye...
research
09/15/2022

On the detrimental effect of invariances in the likelihood for variational inference

Variational Bayesian posterior inference often requires simplifying appr...
research
02/12/2019

Gaussian Mean Field Regularizes by Limiting Learned Information

Variational inference with a factorized Gaussian posterior estimate is a...
research
02/07/2020

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Variational Bayesian Inference is a popular methodology for approximatin...
research
07/10/2023

Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference

We provide a rigorous analysis of training by variational inference (VI)...
research
09/19/2012

Variational Inference in Nonconjugate Models

Mean-field variational methods are widely used for approximate posterior...

Please sign up or login with your details

Forgot password? Click here to reset