HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects

by   Wei Xie, et al.
Southeast University

We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual features between the hand and the object are entangled, and the reconstruction of the interaction area deformation is difficult. To tackle this challenge, we propose a method to annotate our captured data. Our key idea is to collaborate with estimated hand features to guide the object global pose estimation, and then optimize the deformation process of the object by analyzing the relationship between the hand and the object. Through comprehensive evaluation, the proposed method can reconstruct interactive motions of hands and deformable objects with high quality. HMDO currently consists of 21600 frames over 12 sequences. In the future, this dataset could boost the research of learning-based reconstruction of deformable interaction scenes.


page 1

page 3

page 4

page 5

page 6

page 8

page 9

page 10


Robust Shape Estimation for 3D Deformable Object Manipulation

Existing shape estimation methods for deformable object manipulation suf...

3D Reconstruction of Deformable Revolving Object under Heavy Hand Interaction

We reconstruct 3D deformable object through time, in the context of a li...

Textureless Deformable Surface Reconstruction with Invisible Markers

Reconstructing and tracking deformable surface with little or no texture...

InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction

Humans constantly interact with daily objects to accomplish tasks. To un...

SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction

Recent hand-object interaction datasets show limited real object variabi...

Coverage Optimization of Camera Network for Continuous Deformable Object

In this paper, a deformable object is considered for cameras deployment ...

Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution

Estimating the pose and shape of hands and objects under interaction fin...

Please sign up or login with your details

Forgot password? Click here to reset