Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution

by   Hyojoon Park, et al.

We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme.


page 1

page 5

page 7

page 9

page 10

page 15

page 16


Some medical applications of example-based super-resolution

Example-based super-resolution (EBSR) reconstructs a high-resolution ima...

Semi-Supervised Super-Resolution

Super-Resolution is the technique to improve the quality of a low-resolu...

Face Super-Resolution Guided by 3D Facial Priors

State-of-the-art face super-resolution methods employ deep convolutional...

SwiftSRGAN – Rethinking Super-Resolution for Efficient and Real-time Inference

In recent years, there have been several advancements in the task of ima...

Simulation-based Lidar Super-resolution for Ground Vehicles

We propose a methodology for lidar super-resolution with ground vehicles...

Neural Volumetric Blendshapes: Computationally Efficient Physics-Based Facial Blendshapes

Computationally weak systems and demanding graphical applications are st...

Please sign up or login with your details

Forgot password? Click here to reset