Training Deep Energy-Based Models with f-Divergence Minimization

by   Lantao Yu, et al.

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.


page 30

page 31

page 32

page 33

page 34

page 35

page 36

page 37


KALE: When Energy-Based Learning Meets Adversarial Training

Legendre duality provides a variational lower-bound for the Kullback-Lei...

Symmetric-Approximation Energy-Based Estimation of Distribution (SEED): A Continuous Optimization Algorithm

Estimation of Distribution Algorithms (EDAs) maintain and iteratively up...

Spread Divergences

For distributions p and q with different support, the divergence general...

Generalizing Energy-based Generative ConvNets from Particle Evolution Perspective

Compared with Generative Adversarial Networks (GAN), the Energy-Based ge...

Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning

We propose a simple algorithm to train stochastic neural networks to dra...

Efficient Training of Energy-Based Models Using Jarzynski Equality

Energy-based models (EBMs) are generative models inspired by statistical...

Code Repositories


Code for "Training Deep Energy-Based Models with f-Divergence Minimization" ICML 2020

view repo

Please sign up or login with your details

Forgot password? Click here to reset