Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

by   Junxing Hu, et al.

Reconstructing hand-held objects from monocular RGB images is an appealing yet challenging task. In this task, contacts between hands and objects provide important cues for recovering the 3D geometry of the hand-held objects. Though recent works have employed implicit functions to achieve impressive progress, they ignore formulating contacts in their frameworks, which results in producing less realistic object meshes. In this work, we explore how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. Our method consists of two components: explicit contact prediction and implicit shape reconstruction. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. The part-level and vertex-level graph-based transformers are cascaded and jointly learned in a coarse-to-fine manner for more accurate contact probabilities. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space and leverage diffused contact probabilities to construct the implicit neural representation for the manipulated object. Benefiting from estimating the interaction patterns between the hand and the object, our method can reconstruct more realistic object meshes, especially for object parts that are in contact with hands. Extensive experiments on challenging benchmarks show that the proposed method outperforms the current state of the arts by a great margin.


page 8

page 9

page 15

page 16

page 17


Articulated Objects in Free-form Hand Interaction

We use our hands to interact with and to manipulate objects. Articulated...

Reconstructing Hand-Held Objects from Monocular Video

This paper presents an approach that reconstructs a hand-held object fro...

Stability-driven Contact Reconstruction From Monocular Color Images

Physical contact provides additional constraints for hand-object state r...

Nonrigid Object Contact Estimation With Regional Unwrapping Transformer

Acquiring contact patterns between hands and nonrigid objects is a commo...

gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction

Signed distance functions (SDFs) is an attractive framework that has rec...

Unsupervised Learning of 3D Object Categories from Videos in the Wild

Our goal is to learn a deep network that, given a small number of images...

DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field

Reconstructing hand-held objects from a single RGB image is an important...

Please sign up or login with your details

Forgot password? Click here to reset