Active 3D Shape Reconstruction from Vision and Touch

by   Edward J. Smith, et al.

Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. In active touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limited by the lack of frameworks for shape exploration. In this paper, we focus on this problem and introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration. Our framework enables the development of the first fully data-driven solutions to active touch on top of learned models for object understanding. Our experiments show the benefits of such solutions in the task of 3D shape understanding where our models consistently outperform natural baselines. We provide our framework as a tool to foster future research in this direction.


page 9

page 21


Touch if it's transparent! ACTOR: Active Tactile-based Category-Level Transparent Object Reconstruction

Accurate shape reconstruction of transparent objects is a challenging ta...

Learning efficient haptic shape exploration with a rigid tactile sensor array

Haptic exploration is a key skill for both robots and humans to discrimi...

3D Shape Reconstruction from Vision and Touch

When a toddler is presented a new toy, their instinctual behaviour is to...

Data-Driven Measurement Models for Active Localization in Sparse Environments

We develop an algorithm to explore an environment to generate a measurem...

Boundary Cues for 3D Object Shape Recovery

Early work in computer vision considered a host of geometric cues for bo...

Active Visuo-Tactile Interactive Robotic Perception for Accurate Object Pose Estimation in Dense Clutter

This work presents a novel active visuo-tactile based framework for robo...

Semi-Supervised Disentanglement of Tactile Contact Geometry from Sliding-Induced Shear

The sense of touch is fundamental to human dexterity. When mimicked in r...

Please sign up or login with your details

Forgot password? Click here to reset