Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

by   Kai Mo, et al.
Tsinghua University

Sequential multi-step cloth manipulation is a challenging problem in robotic manipulation, requiring a robot to perceive the cloth state and plan a sequence of chained actions leading to the desired state. Most previous works address this problem in a goal-conditioned way, and goal observation must be given for each specific task and cloth configuration, which is not practical and efficient. Thus, we present a novel multi-step cloth manipulation planning framework named Foldformer. Foldformer can complete similar tasks with only a general demonstration and utilize a space-time attention mechanism to capture the instruction information behind this demonstration. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks and show that Foldsformer significantly outperforms state-of-the-art approaches in simulation. Foldformer can complete multi-step cloth manipulation tasks even when configurations of the cloth (e.g., size and pose) vary from configurations in the general demonstrations. Furthermore, our approach can be transferred from simulation to the real world without additional training or domain randomization. Despite training on rectangular clothes, we also show that our approach can generalize to unseen cloth shapes (T-shirts and shorts). Videos and source code are available at:


page 1

page 3

page 6

page 7


VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Robotic fabric manipulation has applications in cloth and cable manageme...

Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics

Vision-based deformable object manipulation is a challenging problem in ...

FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

We address the problem of goal-directed cloth manipulation, a challengin...

VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

Robotic fabric manipulation has applications in home robotics, textiles,...

Augmentation for Learning From Demonstration with Environmental Constraints

We introduce a Learning from Demonstration (LfD) approach for contact-ri...

Conditional Visual Servoing for Multi-Step Tasks

Visual Servoing has been effectively used to move a robot into specific ...

Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation

Automating garment manipulation is challenging due to extremely high var...

Code Repositories


Code for the paper "Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention" (RA-L)

view repo

Please sign up or login with your details

Forgot password? Click here to reset