BreGMN: scaled-Bregman Generative Modeling Networks

by   Akash Srivastava, et al.

The family of f-divergences is ubiquitously applied to generative modeling in order to adapt the distribution of the model to that of the data. Well-definedness of f-divergences, however, requires the distributions of the data and model to overlap completely in every time step of training. As a result, as soon as the support of distributions of data and model contain non-overlapping portions, gradient based training of the corresponding model becomes hopeless. Recent advances in generative modeling are full of remedies for handling this support mismatch problem: key ideas include either modifying the objective function to integral probability measures (IPMs) that are well-behaved even on disjoint probabilities, or optimizing a well-behaved variational lower bound instead of the true objective. We, on the other hand, establish that a complete change of the objective function is unnecessary, and instead an augmentation of the base measure of the problematic divergence can resolve the issue. Based on this observation, we propose a generative model which leverages the class of Scaled Bregman Divergences and generalizes both f-divergences and Bregman divergences. We analyze this class of divergences and show that with the appropriate choice of base measure it can resolve the support mismatch problem and incorporate geometric information. Finally, we study the performance of the proposed method and demonstrate promising results on MNIST, CelebA and CIFAR-10 datasets.


Generative Mixture of Networks

A generative model based on training deep architectures is proposed. The...

On the Convergence of the ELBO to Entropy Sums

The variational lower bound (a.k.a. ELBO or free energy) is the central ...

On the Fisher-Rao Gradient of the Evidence Lower Bound

This article studies the Fisher-Rao gradient, also referred to as the na...

Neural Sequence Model Training via α-divergence Minimization

We propose a new neural sequence model training method in which the obje...

KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

We study the gradient flow for a relaxed approximation to the Kullback-L...

Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning

Finding general evaluation metrics for unsupervised representation learn...

Please sign up or login with your details

Forgot password? Click here to reset