iEdit: Localised Text-guided Image Editing with Weak Supervision

by   Rumeysa Bodur, et al.

Diffusion models (DMs) can generate realistic images with text guidance using large-scale datasets. However, they demonstrate limited controllability in the output space of the generated images. We propose a novel learning method for text-guided image editing, namely , that generates images conditioned on a source image and a textual edit prompt. As a fully-annotated dataset with target images does not exist, previous approaches perform subject-specific fine-tuning at test time or adopt contrastive learning without a target image, leading to issues on preserving the fidelity of the source image. We propose to automatically construct a dataset derived from LAION-5B, containing pseudo-target images with their descriptive edit prompts given input image-caption pairs. This dataset gives us the flexibility of introducing a weakly-supervised loss function to generate the pseudo-target image from the latent noise of the source image conditioned on the edit prompt. To encourage localised editing and preserve or modify spatial structures in the image, we propose a loss function that uses segmentation masks to guide the editing during training and optionally at inference. Our model is trained on the constructed dataset with 200K samples and constrained GPU resources. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.


page 1

page 6

page 7


PRedItOR: Text Guided Image Editing with Diffusion Prior

Diffusion models have shown remarkable capabilities in generating high q...

UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model on a Single Image

We present UniTune, a simple and novel method for general text-driven im...

Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models

Recently large-scale language-image models (e.g., text-guided diffusion ...

From Sparse to Precise: A Practical Editing Approach for Intracardiac Echocardiography Segmentation

Accurate and safe catheter ablation procedures for patients with atrial ...

Deep Learning-based Forgery Attack on Document Images

With the ongoing popularization of online services, the digital document...

Incorporating Pseudo-Parallel Data for Quantifiable Sequence Editing

In the task of quantifiable sequence editing (QuaSE), a model needs to e...

Editing Implicit Assumptions in Text-to-Image Diffusion Models

Text-to-image diffusion models often make implicit assumptions about the...

Please sign up or login with your details

Forgot password? Click here to reset