Assessing Generative Models via Precision and Recall

by   Mehdi S. M. Sajjadi, et al.

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as Fréchet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and the Variational Autoencoder. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.


page 8

page 12

page 14


Toward a Generalization Metric for Deep Generative Models

Measuring the generalization capacity of Deep Generative Models (DGMs) i...

Training Normalizing Flows with the Precision-Recall Divergence

Generative models can have distinct mode of failures like mode dropping ...

Improved Precision and Recall Metric for Assessing Generative Models

The ability to evaluate the performance of a computational model is a vi...

Revisiting Precision and Recall Definition for Generative Model Evaluation

In this article we revisit the definition of Precision-Recall (PR) curve...

Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows

Achieving a balance between image quality (precision) and diversity (rec...

Pros and Cons of GAN Evaluation Measures: New Developments

This work is an update of a previous paper on the same topic published a...

Evaluating Generative Models Using Divergence Frontiers

Despite the tremendous progress in the estimation of generative models, ...

Please sign up or login with your details

Forgot password? Click here to reset