DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity

by   Melissa Hall, et al.

The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, we introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparities, an important step towards building responsible visual content creation systems. We use our proposed indicators to analyze potential geographic biases in state-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at a cost to prompt-consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone.


page 9

page 10

page 13

page 24

page 26

page 27

page 28

page 29


Fairness Indicators for Systematic Assessments of Visual Feature Extractors

Does everyone equally benefit from computer vision systems? Answers to t...

Breaking Barriers to Creative Expression: Co-Designing and Implementing an Accessible Text-to-Image Interface

Text-to-image generation models have grown in popularity due to their ab...

A Prompt Log Analysis of Text-to-Image Generation Systems

Recent developments in large language models (LLM) and generative AI hav...

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Generating images from textual descriptions has gained a lot of attentio...

Network Fusion for Content Creation with Conditional INNs

Artificial Intelligence for Content Creation has the potential to reduce...

Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation

Research in Image Generation has recently made significant progress, par...

TIAM – A Metric for Evaluating Alignment in Text-to-Image Generation

The progress in the generation of synthetic images has made it crucial t...

Please sign up or login with your details

Forgot password? Click here to reset