StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pretrained StyleGAN

by   Fei Yin, et al.

One-shot talking face generation aims at synthesizing a high-quality talking face video from an arbitrary portrait image, driven by a video or an audio segment. One challenging quality factor is the resolution of the output video: higher resolution conveys more details. In this work, we investigate the latent feature space of a pre-trained StyleGAN and discover some excellent spatial transformation properties. Upon the observation, we explore the possibility of using a pre-trained StyleGAN to break through the resolution limit of training datasets. We propose a novel unified framework based on a pre-trained StyleGAN that enables a set of powerful functionalities, i.e., high-resolution video generation, disentangled control by driving video or audio, and flexible face editing. Our framework elevates the resolution of the synthesized talking face to 1024*1024 for the first time, even though the training dataset has a lower resolution. We design a video-based motion generation module and an audio-based one, which can be plugged into the framework either individually or jointly to drive the video generation. The predicted motion is used to transform the latent features of StyleGAN for visual animation. To compensate for the transformation distortion, we propose a calibration network as well as a domain loss to refine the features. Moreover, our framework allows two types of facial editing, i.e., global editing via GAN inversion and intuitive editing based on 3D morphable models. Comprehensive experiments show superior video quality, flexible controllability, and editability over state-of-the-art methods.


page 4

page 8

page 10

page 11

page 12

page 13

page 14

page 15


IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis

Existing 3D-aware facial generation methods face a dilemma in quality ve...

Sound-Guided Semantic Video Generation

The recent success in StyleGAN demonstrates that pre-trained StyleGAN la...

FacialFilmroll: High-resolution multi-shot video editing

We present FacialFilmroll, a solution for spatially and temporally consi...

Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor

This paper proposes a video editor based on OpenShot with several state-...

High-Resolution Talking Face Generation via Mutual Information Approximation

Given an arbitrary speech clip and a facial image, talking face generati...

High Resolution Face Age Editing

Face age editing has become a crucial task in film post-production, and ...

High-resolution Face Swapping via Latent Semantics Disentanglement

We present a novel high-resolution face swapping method using the inhere...

Code Repositories


A framework for high-resolution editable talking face generation

view repo

Please sign up or login with your details

Forgot password? Click here to reset