High-Fidelity and Freely Controllable Talking Head Video Generation

by   Yue Gao, et al.

Talking head generation is to generate video based on a given source identity and target motion. However, current methods face several challenges that limit the quality and controllability of the generated videos. First, the generated face often has unexpected deformation and severe distortions. Second, the driving image does not explicitly disentangle movement-relevant information, such as poses and expressions, which restricts the manipulation of different attributes during generation. Third, the generated videos tend to have flickering artifacts due to the inconsistency of the extracted landmarks between adjacent frames. In this paper, we propose a novel model that produces high-fidelity talking head videos with free control over head pose and expression. Our method leverages both self-supervised learned landmarks and 3D face model-based landmarks to model the motion. We also introduce a novel motion-aware multi-scale feature alignment module to effectively transfer the motion without face distortion. Furthermore, we enhance the smoothness of the synthesized talking head videos with a feature context adaptation and propagation module. We evaluate our model on challenging datasets and demonstrate its state-of-the-art performance. More information is available at https://yuegao.me/PECHead.


page 1

page 4

page 6

page 7

page 8


One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

Talking head generation aims to generate faces that maintain the identit...

Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation

Talking head video generation aims to animate a human face in a still im...

Controllable One-Shot Face Video Synthesis With Semantic Aware Prior

The one-shot talking-head synthesis task aims to animate a source image ...

Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation

This paper presents a novel approach for generating 3D talking heads fro...

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

Synthesizing high-fidelity head avatars is a central problem for compute...

CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generation

Recently, talking face generation has drawn ever-increasing attention fr...

MetaHead: An Engine to Create Realistic Digital Head

Collecting and labeling training data is one important step for learning...

Please sign up or login with your details

Forgot password? Click here to reset