Robust, Automated, and Accurate Black-box Variational Inference

by   Manushi Welandawe, et al.

Black-box variational inference (BBVI) now sees widespread use in machine learning and statistics as a fast yet flexible alternative to Markov chain Monte Carlo methods for approximate Bayesian inference. However, stochastic optimization methods for BBVI remain unreliable and require substantial expertise and hand-tuning to apply effectively. In this paper, we propose Robust, Automated, and Accurate BBVI (RAABBVI), a framework for reliable BBVI optimization. RAABBVI is based on rigorously justified automation techniques, includes just a small number of intuitive tuning parameters, and detects inaccurate estimates of the optimal variational approximation. RAABBVI adaptively decreases the learning rate by detecting convergence of the fixed–learning-rate iterates, then estimates the symmetrized Kullback–Leiber (KL) divergence between the current variational approximation and the optimal one. It also employs a novel optimization termination criterion that enables the user to balance desired accuracy against computational cost by comparing (i) the predicted relative decrease in the symmetrized KL divergence if a smaller learning were used and (ii) the predicted computation required to converge with the smaller learning rate. We validate the robustness and accuracy of RAABBVI through carefully designed simulation studies and on a diverse set of real-world model and data examples.


page 1

page 2

page 3

page 4


Black Box Variational Inference

Variational inference has become a widely used method to approximate pos...

Robust, Accurate Stochastic Optimization for Variational Inference

We consider the problem of fitting variational posterior approximations ...

A Dual Control Variate for doubly stochastic optimization and black-box variational inference

In this paper, we aim at reducing the variance of doubly stochastic opti...

Variational Inference via χ-Upper Bound Minimization

Variational inference (VI) is widely used as an efficient alternative to...

Perturbative Black Box Variational Inference

Black box variational inference (BBVI) with reparameterization gradients...

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

Uncertainty computation in deep learning is essential to design robust a...

Implicit copula variational inference

Key to effective generic, or "black-box", variational inference is the s...

Please sign up or login with your details

Forgot password? Click here to reset