DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

by   Bao Thach, et al.

Applications in fields ranging from home care to warehouse fulfillment to surgical assistance require robots to reliably manipulate the shape of 3D deformable objects. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the manipulated object and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn a visual servo controller that computes the desired robot end-effector action to iteratively deform the object toward the target shape. We demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training. Crucially, using DeformerNet, the robot successfully accomplishes three surgical sub-tasks: retraction (moving tissue aside to access a site underneath it), tissue wrapping (a sub-task in procedures like aortic stent placements), and connecting two tubular pieces of tissue (a sub-task in anastomosis).


page 6

page 9

page 12

page 14

page 15

page 16

page 17

page 18


Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

If robots could reliably manipulate the shape of 3D deformable objects, ...

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

In this paper, we propose a novel approach to 3D deformable object manip...

Lattice-based shape tracking and servoing of elastic objects

In this paper, we propose a general unified tracking-servoing approach f...

Robotic Dough Shaping

We address the problem of shaping a piece of dough-like deformable mater...

Towards Latent Space Based Manipulation of Elastic Rods using Autoencoder Models and Robust Centerline Extractions

The automatic shape control of deformable objects is a challenging (and ...

Learn the Manipulation of Deformable Objects Using Tangent Space Point Set Registration

Point set registration is a powerful method that enables robots to manip...

Adaptive Shape Servoing of Elastic Rods using Parameterized Regression Features and Auto-Tuning Motion Controls

In this paper, we present a new vision-based method to control the shape...

Please sign up or login with your details

Forgot password? Click here to reset