DiffFace: Diffusion-based Face Swapping with Facial Guidance

by   Kihong Kim, et al.

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.


page 1

page 7

page 8

page 14

page 15

page 16

page 17

page 18


SimSwap: An Efficient Framework For High Fidelity Face Swapping

We propose an efficient framework, called Simple Swap (SimSwap), aiming ...

MFIM: Megapixel Facial Identity Manipulation

Face swapping is a task that changes a facial identity of a given image ...

Face Animation with an Attribute-Guided Diffusion Model

Face animation has achieved much progress in computer vision. However, p...

RobustSwap: A Simple yet Robust Face Swapping Model against Attribute Leakage

Face swapping aims at injecting a source image's identity (i.e., facial ...

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection

Non-parametric face modeling aims to reconstruct 3D face only from image...

Investigating Prompt Engineering in Diffusion Models

With the spread of the use of Text2Img diffusion models such as DALL-E 2...

Reinforced Disentanglement for Face Swapping without Skip Connection

The SOTA face swap models still suffer the problem of either target iden...

Please sign up or login with your details

Forgot password? Click here to reset