Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

by   Bao Thach, et al.

If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn to define a visual servo controller that provides Cartesian pose changes to the robot end-effector causing the object to deform towards its target shape. Crucially, we demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training and outperforms comparison methods for both the generic shape control and the surgical task of retraction.


page 1

page 3

page 4

page 6


DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

Applications in fields ranging from home care to warehouse fulfillment t...

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

In this paper, we propose a novel approach to 3D deformable object manip...

Lattice-based shape tracking and servoing of elastic objects

In this paper, we propose a general unified tracking-servoing approach f...

A controller for reaching and unveiling a partially occluded object of interest with an eye-in-hand robot

In this work, a control scheme for approaching and unveiling a partially...

Robotic Dough Shaping

We address the problem of shaping a piece of dough-like deformable mater...

Sequential Topological Representations for Predictive Models of Deformable Objects

Deformable objects present a formidable challenge for robotic manipulati...

Large Scale Novel Object Discovery in 3D

We present a method for discovering objects in 3D point clouds from sens...

Please sign up or login with your details

Forgot password? Click here to reset