One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

by   Weichuang Li, et al.

Talking head generation aims to generate faces that maintain the identity information of the source image and imitate the motion of the driving image. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from face distortion when large head rotations are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of identity and expression is not so desirable, especially for novel-view synthesis. In this paper, we propose HiDe-NeRF, which achieves high-fidelity and free-view talking-head synthesis. Drawing on the recently proposed Deformable Neural Radiance Fields, HiDe-NeRF represents the 3D dynamic scene into a canonical appearance field and an implicit deformation field, where the former comprises the canonical source face and the latter models the driving pose and expression. In particular, we improve fidelity from two aspects: (i) to enhance identity expressiveness, we design a generalized appearance module that leverages multi-scale volume features to preserve face shape and details; (ii) to improve expression preciseness, we propose a lightweight deformation module that explicitly decouples the pose and expression to enable precise expression modeling. Extensive experiments demonstrate that our proposed approach can generate better results than previous works. Project page:


page 6

page 8

page 14

page 15

page 16

page 17

page 18

page 19


High-Fidelity and Freely Controllable Talking Head Video Generation

Talking head generation is to generate video based on a given source ide...

Deformable Model Driven Neural Rendering for High-fidelity 3D Reconstruction of Human Heads Under Low-View Settings

We propose a robust method for learning neural implicit functions that c...

Generalizable One-shot Neural Head Avatar

We present a method that reconstructs and animates a 3D head avatar from...

Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation

Talking head video generation aims to animate a human face in a still im...

Controllable One-Shot Face Video Synthesis With Semantic Aware Prior

The one-shot talking-head synthesis task aims to animate a source image ...

FNeVR: Neural Volume Rendering for Face Animation

Face animation, one of the hottest topics in computer vision, has achiev...

High-Fidelity Eye Animatable Neural Radiance Fields for Human Face

Face rendering using neural radiance fields (NeRF) is a rapidly developi...

Please sign up or login with your details

Forgot password? Click here to reset