On the De-duplication of LAION-2B

by   Ryan Webster, et al.
Université de Caen Normandie

Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.


DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models

With recent advancements in diffusion models, users can generate high-qu...

HoloDiffusion: Training a 3D Diffusion Model using 2D Images

Diffusion models have emerged as the best approach for generative modeli...

Detecting Images Generated by Diffusers

This paper explores the task of detecting images generated by text-to-im...

Fake it till you make it: Learning(s) from a synthetic ImageNet clone

Recent large-scale image generation models such as Stable Diffusion have...

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

A diffusion model learns to predict a vector field of gradients. We prop...

Measuring the Success of Diffusion Models at Imitating Human Artists

Modern diffusion models have set the state-of-the-art in AI image genera...

Automated Analysis, Reporting, and Archiving for Robotic Nondestructive Assay of Holdup Deposits

To decommission deactivated gaseous diffusion enrichment facilities, mil...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset