Localizing Object-level Shape Variations with Text-to-Image Diffusion Models

03/20/2023
by   Or Patashnik, et al.
0

Text-to-image models give rise to workflows which often begin with an exploration step, where users sift through a large collection of generated images. The global nature of the text-to-image generation process prevents users from narrowing their exploration to a particular object in the image. In this paper, we present a technique to generate a collection of images that depicts variations in the shape of a specific object, enabling an object-level shape exploration process. Creating plausible variations is challenging as it requires control over the shape of the generated object while respecting its semantics. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object's shape. We introduce a prompt-mixing technique that switches between prompts along the denoising process to attain a variety of shape choices. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers. Moreover, we show that these localization techniques are general and effective beyond the scope of generating object variations. Extensive results and comparisons demonstrate the effectiveness of our method in generating object variations, and the competence of our localization techniques.

READ FULL TEXT

page 1

page 6

page 7

page 8

page 12

page 14

page 15

page 16

research
05/30/2023

Real-World Image Variation by Aligning Diffusion Inversion Chain

Recent diffusion model advancements have enabled high-fidelity images to...
research
02/05/2023

Mixture of Diffusers for scene composition and high resolution image generation

Diffusion methods have been proven to be very effective to generate imag...
research
03/01/2023

Collage Diffusion

Text-conditional diffusion models generate high-quality, diverse images....
research
06/04/2023

Detector Guidance for Multi-Object Text-to-Image Generation

Diffusion models have demonstrated impressive performance in text-to-ima...
research
12/01/2022

Shape-Guided Diffusion with Inside-Outside Attention

Shape can specify key object constraints, yet existing text-to-image dif...
research
03/21/2023

Vox-E: Text-guided Voxel Editing of 3D Objects

Large scale text-guided diffusion models have garnered significant atten...
research
08/04/2022

DropKey

In this paper, we focus on analyzing and improving the dropout technique...

Please sign up or login with your details

Forgot password? Click here to reset