The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models

by   Shengjia Zhao, et al.
Stanford University

A variety of learning objectives have been proposed for training latent variable generative models. We show that many of them, including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial autoencoders, AVB, AS-VAE and InfoVAE, are Lagrangian duals of the same primal optimization problem, corresponding to different settings of the Lagrange multipliers. The primal problem optimizes the mutual information between latent and visible variables, subject to the constraints of accurately modeling the data distribution and performing correct amortized inference. Based on this observation, we provide an exhaustive characterization of the statistical and computational trade-offs made by all the training objectives in this class of Lagrangian duals. Next, we propose a dual optimization method where we optimize model parameters as well as the Lagrange multipliers. This method achieves Pareto near-optimal solutions in terms of optimizing information and satisfying the consistency constraints.


page 1

page 2

page 3

page 4


An Information-Theoretic Analysis of Deep Latent-Variable Models

We present an information-theoretic framework for understanding trade-of...

Lagging Inference Networks and Posterior Collapse in Variational Autoencoders

The variational autoencoder (VAE) is a popular combination of deep laten...

Information bottleneck through variational glasses

Information bottleneck (IB) principle [1] has become an important elemen...

Improve variational autoEncoder with auxiliary softmax multiclassifier

As a general-purpose generative model architecture, VAE has been widely ...

Primal-Dual Wasserstein GAN

We introduce Primal-Dual Wasserstein GAN, a new learning algorithm for b...

A Lagrangian Duality Approach to Active Learning

We consider the batch active learning problem, where only a subset of th...

Disentangled Information Bottleneck

The information bottleneck (IB) method is a technique for extracting inf...

Please sign up or login with your details

Forgot password? Click here to reset