On Batch Normalisation for Approximate Bayesian Inference

by   Jishnu Mukhoti, et al.
University of Oxford

We study batch normalisation in the context of variational inference methods in Bayesian neural networks, such as mean-field or MC Dropout. We show that batch-normalisation does not affect the optimum of the evidence lower bound (ELBO). Furthermore, we study the Monte Carlo Batch Normalisation (MCBN) algorithm, proposed as an approximate inference technique parallel to MC Dropout, and show that for larger batch sizes, MCBN fails to capture epistemic uncertainty. Finally, we provide insights into what is required to fix this failure, namely having to view the mini-batch size as a variational parameter in MCBN. We comment on the asymptotics of the ELBO with respect to this variational parameter, showing that as dataset size increases towards infinity, the batch-size must increase towards infinity as well for MCBN to be a valid approximate inference technique.


page 1

page 2

page 3

page 4


Is MC Dropout Bayesian?

MC Dropout is a mainstream "free lunch" method in medical imaging for ap...

Alpha-Divergences in Variational Dropout

We investigate the use of alternative divergences to Kullback-Leibler (K...

Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network

Two main obstacles preventing the widespread adoption of variational Bay...

Parallel Dither and Dropout for Regularising Deep Neural Networks

Effective regularisation during training can mean the difference between...

GFlowOut: Dropout with Generative Flow Networks

Bayesian Inference offers principled tools to tackle many critical probl...

Scalable approximate inference for state space models with normalising flows

By exploiting mini-batch stochastic gradient optimisation, variational i...

When random initializations help: a study of variational inference for community detection

Variational approximation has been widely used in large-scale Bayesian i...

Please sign up or login with your details

Forgot password? Click here to reset