VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

by   Ryan Hoque, et al.

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We build upon the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks with a single goal-conditioned policy. We extend our earlier work on VisuoSpatial Foresight (VSF), which learns visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. In this earlier work, we evaluated VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. A key finding was that depth sensing significantly improves performance: RGBD data yields an 80 in simulation over pure RGB data. In this work, we vary 4 components of VSF, including data generation, the choice of visual dynamics model, cost function, and optimization procedure. Results suggest that training visual dynamics models using longer, corner-based actions can improve the efficiency of fabric folding by 76 could not previously perform with 90 supplementary material are available at


page 3

page 10

page 15

page 16

page 18

page 22

page 26

page 28


VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Robotic fabric manipulation has applications in cloth and cable manageme...

Deep Imitation Learning of Sequential Fabric Smoothing Policies

Sequential pulling policies to flatten and smooth fabrics have applicati...

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Sequential multi-step cloth manipulation is a challenging problem in rob...

Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

Automation of surgical tasks using cable-driven robots is challenging du...

Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot

Recent advances in depth-sensing have significantly increased accuracy, ...

Robotic Fabric Flattening with Wrinkle Direction Detection

Deformable Object Manipulation (DOM) is an important field of research a...

Learning Robot Geometry as Distance Fields: Applications to Whole-body Manipulation

In this work, we propose to learn robot geometry as distance fields (RDF...

Please sign up or login with your details

Forgot password? Click here to reset