Unbiased Gradient Estimation for Variational Auto-Encoders using Coupled Markov Chains

10/05/2020
by   Francisco J. R. Ruiz, et al.
9

The variational auto-encoder (VAE) is a deep latent variable model that has two neural networks in an autoencoder-like architecture; one of them parameterizes the model's likelihood. Fitting its parameters via maximum likelihood is challenging since the computation of the likelihood involves an intractable integral over the latent space; thus the VAE is trained instead by maximizing a variational lower bound. Here, we develop a maximum likelihood training scheme for VAEs by introducing unbiased gradient estimators of the log-likelihood. We obtain the unbiased estimators by augmenting the latent space with a set of importance samples, similarly to the importance weighted auto-encoder (IWAE), and then constructing a Markov chain Monte Carlo (MCMC) coupling procedure on this augmented space. We provide the conditions under which the estimators can be computed in finite time and have finite variance. We demonstrate experimentally that VAEs fitted with unbiased estimators exhibit better predictive performance on three image datasets.

READ FULL TEXT
05/29/2018

Hamiltonian Variational Auto-Encoder

Variational Auto-Encoders (VAEs) have become very popular techniques to ...
12/29/2020

Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler

Due to the intractable partition function, training energy-based models ...
03/03/2019

Variational Auto-Decoder

Learning a generative model from partial data (data with missingness) is...
03/01/2021

Latent linear dynamics in spatiotemporal medical data

Spatiotemporal imaging is common in medical imaging, with applications i...
03/11/2019

Consistency of the maximum likelihood and variational estimators in a dynamic stochastic block model

We consider a dynamic version of the stochastic block model, in which th...
04/30/2020

Maximum likelihood estimation of the Fisher-Bingham distribution via efficient calculation of its normalizing constant

This paper proposes an efficient numerical integration formula to comput...
03/25/2022

Efficient-VDVAE: Less is more

Hierarchical VAEs have emerged in recent years as a reliable option for ...

Please sign up or login with your details

Forgot password? Click here to reset