OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction

by   Wenbin Lin, et al.
Tsinghua University

RGBD-based real-time dynamic 3D reconstruction suffers from inaccurate inter-frame motion estimation as errors may accumulate with online tracking. This problem is even more severe for single-view-based systems due to strong occlusions. Based on these observations, we propose OcclusionFusion, a novel method to calculate occlusion-aware 3D motion to guide the reconstruction. In our technique, the motion of visible regions is first estimated and combined with temporal information to infer the motion of the occluded regions through an LSTM-involved graph neural network. Furthermore, our method computes the confidence of the estimated motion by modeling the network output with a probabilistic model, which alleviates untrustworthy motions and enables robust tracking. Experimental results on public datasets and our own recorded data show that our technique outperforms existing single-view-based real-time methods by a large margin. With the reduction of the motion errors, the proposed technique can handle long and challenging motion sequences. Please check out the project page for sequence results: https://wenbin-lin.github.io/OcclusionFusion.


page 1

page 6

page 7

page 8

page 12

page 13


ArticulatedFusion: Real-time Reconstruction of Motion, Geometry and Segmentation Using a Single Depth Camera

This paper proposes a real-time dynamic scene reconstruction method capa...

BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

We present a near real-time method for 6-DoF tracking of an unknown obje...

Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing

Video snapshot compressive imaging (SCI) utilizes a 2D detector to captu...

Development of a Self-Calibrated Motion Capture System by Nonlinear Trilateration of Multiple Kinects v2

In this paper, a Kinect-based distributed and real-time motion capture s...

Tracking Everything Everywhere All at Once

We present a new test-time optimization method for estimating dense and ...

Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

In this paper we propose a novel deep neural network that is able to joi...

A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions

Motion estimation is one of the core challenges in computer vision. With...

Please sign up or login with your details

Forgot password? Click here to reset