Reliable Fidelity and Diversity Metrics for Generative Models

by   Muhammad Ferjad Naeem, et al.

Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code:


page 1

page 2

page 3

page 4

page 5

page 6

page 8

page 9


Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models

Assessing the fidelity and diversity of the generative model is a diffic...

Barcode Method for Generative Model Evaluation driven by Topological Data Analysis

Evaluating the performance of generative models in image synthesis is a ...

Unsupervised evaluation of GAN sample quality: Introducing the TTJac Score

Evaluation metrics are essential for assessing the performance of genera...

Efficient Graph-Friendly COCO Metric Computation for Train-Time Model Evaluation

Evaluating the COCO mean average precision (MaP) and COCO recall metrics...

Evaluation Metrics for Conditional Image Generation

We present two new metrics for evaluating generative models in the class...

Reliable and Efficient Image Cropping: A Grid Anchor based Approach

Image cropping aims to improve the composition as well as aesthetic qual...

Code Repositories


Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

view repo


Lots of evaluation metrics for the generative adversarial networks in pytorch

view repo

Please sign up or login with your details

Forgot password? Click here to reset