DeepAI AI Chat
Log In Sign Up

Improving Importance Weighted Auto-Encoders with Annealed Importance Sampling

by   Xinqiang Ding, et al.
University of Michigan
The University of Chicago

Stochastic variational inference with an amortized inference model and the reparameterization trick has become a widely-used algorithm for learning latent variable models. Increasing the flexibility of approximate posterior distributions while maintaining computational tractability is one of the core problems in stochastic variational inference. Two families of approaches proposed to address the problem are flow-based and multisample-based approaches such as importance weighted auto-encoders (IWAE). We introduce a new learning algorithm, the annealed importance weighted auto-encoder (AIWAE), for learning latent variable models. The proposed AIWAE combines multisample-based and flow-based approaches with the annealed importance sampling and its memory cost stays constant when the depth of flows increases. The flow constructed using an annealing process in AIWAE facilitates the exploration of the latent space when the posterior distribution has multiple modes. Through computational experiments, we show that, compared to models trained using the IWAE, AIWAE-trained models are better density models, have more complex posterior distributions and use more latent space representation capacity.


page 1

page 2

page 3

page 4


Improving Explorability in Variational Inference with Annealed Variational Objectives

Despite the advances in the representational capacity of approximate dis...

Importance Weighted Adversarial Variational Autoencoders for Spike Inference from Calcium Imaging Data

The Importance Weighted Auto Encoder (IWAE) objective has been shown to ...

FIS-GAN: GAN with Flow-based Importance Sampling

Generative Adversarial Networks (GAN) training process, in most cases, a...

SentenceMIM: A Latent Variable Language Model

We introduce sentenceMIM, a probabilistic auto-encoder for language mode...

Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencoder

We consider the problem of diversifying automated reply suggestions for ...

Monte Carlo Variational Auto-Encoders

Variational auto-encoders (VAE) are popular deep latent variable models ...

On the Relationship Between Variational Inference and Auto-Associative Memory

In this article, we propose a variational inference formulation of auto-...