MirrorNeRF: One-shot Neural Portrait RadianceField from Multi-mirror Catadioptric Imaging

by   Ziyu Wang, et al.

Photo-realistic neural reconstruction and rendering of the human portrait are critical for numerous VR/AR applications. Still, existing solutions inherently rely on multi-view capture settings, and the one-shot solution to get rid of the tedious multi-view synchronization and calibration remains extremely challenging. In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric imaging so as to enable one-shot photo-realistic human portrait reconstruction and rendering, in a low-cost and casual capture setting. More specifically, we propose a light-weight catadioptric system design with a sphere mirror array to enable diverse ray sampling in the continuous 3D space as well as an effective online calibration for the camera and the mirror array. Our catadioptric imaging system can be easily deployed with a low budget and the casual capture ability for convenient daily usages. We introduce a novel neural warping radiance field representation to learn a continuous displacement field that implicitly compensates for the misalignment due to our flexible system setting. We further propose a density regularization scheme to leverage the inherent geometry information from the catadioptric data in a self-supervision manner, which not only improves the training efficiency but also provides more effective density supervision for higher rendering quality. Extensive experiments demonstrate the effectiveness and robustness of our scheme to achieve one-shot photo-realistic and high-quality appearance free-viewpoint rendering for human portrait scenes.


page 2

page 4

page 7

page 8

page 9

page 10


MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging

Photo-realistic neural reconstruction and rendering of the human portrai...

Convolutional Neural Opacity Radiance Fields

Photo-realistic modeling and rendering of fuzzy objects with complex opa...

Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting

Detailed 3D reconstruction and photo-realistic relighting of digital hum...

Volumetric Capture of Humans with a Single RGBD Camera via Semi-Parametric Learning

Volumetric (4D) performance capture is fundamental for AR/VR content gen...

FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

Novel view synthesis with sparse inputs is a challenging problem for neu...

From NeRFLiX to NeRFLiX++: A General NeRF-Agnostic Restorer Paradigm

Neural radiance fields (NeRF) have shown great success in novel view syn...

Editable Free-viewpoint Video Using a Layered Neural Representation

Generating free-viewpoint videos is critical for immersive VR/AR experie...

Please sign up or login with your details

Forgot password? Click here to reset