RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream

by   Zhuo Su, et al.

High-quality 4D reconstruction of human performance with complex interactions to various objects is essential in real-world scenarios, which enables numerous immersive VR/AR applications. However, recent advances still fail to provide reliable performance reconstruction, suffering from challenging interaction patterns and severe occlusions, especially for the monocular setting. To fill this gap, in this paper, we propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios using only a single RGBD sensor, which combines various data-driven visual and interaction cues to handle the complex interaction patterns and severe occlusions. We propose a semantic-aware scene decoupling scheme to model the occlusions explicitly, with a segmentation refinement and robust object tracking to prevent disentanglement uncertainty and maintain temporal consistency. We further introduce a robust performance capture scheme with the aid of various data-driven cues, which not only enables re-initialization ability, but also models the complex human-object interaction patterns in a data-driven manner. To this end, we introduce a spatial relation prior to prevent implausible intersections, as well as data-driven interaction cues to maintain natural motions, especially for those regions under severe human-object occlusions. We also adopt an adaptive fusion scheme for temporally coherent human-object reconstruction with occlusion analysis and human parsing cue. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality 4D human performance reconstruction under complex human-object interactions whilst still maintaining the lightweight monocular setting.


page 1

page 3

page 9

page 10

page 11

page 12

page 13

page 14


Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

4D reconstruction of human-object interaction is critical for immersive ...

Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream

Convenient 4D modeling of human-object interactions is essential for num...

NeuralFusion: Neural Volumetric Rendering under Human-object Interactions

4D modeling of human-object interactions is critical for numerous applic...

iMapper: Interaction-guided Joint Scene and Human Motion Mapping from Monocular Videos

A long-standing challenge in scene analysis is the recovery of scene arr...

RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects

There have been remarkable successes in computer vision with deep learni...

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

Capturing challenging human motions is critical for numerous application...

Neural Video Portrait Relighting in Real-time via Consistency Modeling

Video portraits relighting is critical in user-facing human photography,...

Please sign up or login with your details

Forgot password? Click here to reset