Editing Implicit Assumptions in Text-to-Image Diffusion Models

03/14/2023
by   Hadas Orgad, et al.
0

Text-to-image diffusion models often make implicit assumptions about the world when generating images. While some assumptions are useful (e.g., the sky is blue), they can also be outdated, incorrect, or reflective of social biases present in the training data. Thus, there is a need to control these assumptions without requiring explicit user input or costly re-training. In this work, we aim to edit a given implicit assumption in a pre-trained diffusion model. Our Text-to-Image Model Editing method, TIME for short, receives a pair of inputs: a "source" under-specified prompt for which the model makes an implicit assumption (e.g., "a pack of roses"), and a "destination" prompt that describes the same setting, but with a specified desired attribute (e.g., "a pack of blue roses"). TIME then updates the model's cross-attention layers, as these layers assign visual meaning to textual tokens. We edit the projection matrices in these layers such that the source prompt is projected close to the destination prompt. Our method is highly efficient, as it modifies a mere 2.2 second. To evaluate model editing approaches, we introduce TIMED (TIME Dataset), containing 147 source and destination prompt pairs from various domains. Our experiments (using Stable Diffusion) show that TIME is successful in model editing, generalizes well for related prompts unseen during editing, and imposes minimal effect on unrelated generations.

READ FULL TEXT

page 1

page 2

page 3

page 5

page 6

page 7

page 8

page 13

research
06/22/2023

Continuous Layout Editing of Single Images with Diffusion Models

Recent advancements in large-scale text-to-image diffusion models have e...
research
07/06/2023

Applying a Color Palette with Local Control using Diffusion Models

We demonstrate two novel editing procedures in the context of fantasy ca...
research
09/01/2023

Iterative Multi-granular Image Editing using Diffusion Models

Recent advances in text-guided image synthesis has dramatically changed ...
research
03/30/2023

PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models

Image editing using diffusion models has witnessed extremely fast-paced ...
research
05/10/2023

iEdit: Localised Text-guided Image Editing with Weak Supervision

Diffusion models (DMs) can generate realistic images with text guidance ...
research
04/05/2023

JPEG Compressed Images Can Bypass Protections Against AI Editing

Recently developed text-to-image diffusion models make it easy to edit o...
research
05/22/2023

The CLIP Model is Secretly an Image-to-Prompt Converter

The Stable Diffusion model is a prominent text-to-image generation model...

Please sign up or login with your details

Forgot password? Click here to reset