Iterative Text-based Editing of Talking-heads Using Neural Retargeting

by   Xinwei Yao, et al.

We present a text-based tool for editing talking-head video that enables an iterative editing workflow. On each iteration users can edit the wording of the speech, further refine mouth motions if necessary to reduce artifacts and manipulate non-verbal aspects of the performance by inserting mouth gestures (e.g. a smile) or changing the overall performance style (e.g. energetic, mumble). Our tool requires only 2-3 minutes of the target actor video and it synthesizes the video for each iteration in about 40 seconds, allowing users to quickly explore many editing possibilities as they iterate. Our approach is based on two key ideas. (1) We develop a fast phoneme search algorithm that can quickly identify phoneme-level subsequences of the source repository video that best match a desired edit. This enables our fast iteration loop. (2) We leverage a large repository of video of a source actor and develop a new self-supervised neural retargeting technique for transferring the mouth motions of the source actor to the target actor. This allows us to work with relatively short target actor videos, making our approach applicable in many real-world editing scenarios. Finally, our refinement and performance controls give users the ability to further fine-tune the synthesized results.


page 2

page 3

page 5

page 6

page 7

page 12

page 13

page 14


Edit-A-Video: Single Video Editing with Object-Aware Consistency

Despite the fact that text-to-video (TTV) model has recently achieved re...

Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts

The text-driven image and video diffusion models have achieved unprecede...

Deep Video Portraits

We present a novel approach that enables photo-realistic re-animation of...

Neural Style-Preserving Visual Dubbing

Dubbing is a technique for translating video content from one language t...

Responsive Action-based Video Synthesis

We propose technology to enable a new medium of expression, where video ...

Manual Post-editing of Automatically Transcribed Speeches from the Icelandic Parliament - Althingi

The design objectives for an automatic transcription system are to produ...

Please sign up or login with your details

Forgot password? Click here to reset