Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers

by   Yixuan Huang, et al.

Objects rarely sit in isolation in everyday human environments. If we want robots to operate and perform tasks in our human environments, they must understand how the objects they manipulate will interact with structural elements of the environment for all but the simplest of tasks. As such, we'd like our robots to reason about how multiple objects and environmental elements relate to one another and how those relations may change as the robot interacts with the world. We examine the problem of predicting inter-object and object-environment relations between previously unseen objects and novel environments purely from partial-view point clouds. Our approach enables robots to plan and execute sequences to complete multi-object manipulation tasks defined from logical relations. This removes the burden of providing explicit, continuous object states as goals to the robot. We explore several different neural network architectures for this task. We find the best performing model to be a novel transformer-based neural network that both predicts object-environment relations and learns a latent-space dynamics function. We achieve reliable sim-to-real transfer without any fine-tuning. Our experiments show that our model understands how changes in observed environmental geometry relate to semantic relations between objects. We show more videos on our website:


page 1

page 6

page 8

page 9

page 11

page 12


Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

Objects rarely sit in isolation in human environments. As such, we'd lik...

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Geometric organization of objects into semantically meaningful arrangeme...

Semantic Linking Maps for Active Visual Object Search

We aim for mobile robots to function in a variety of common human enviro...

Collective Intelligence for Object Manipulation with Mobile Robots

While natural systems often present collective intelligence that allows ...

Improving Object Permanence using Agent Actions and Reasoning

Object permanence in psychology means knowing that objects still exist e...

SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning

Building general-purpose robots to perform an enormous amount of tasks i...

StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects

Robots operating in human environments must be able to rearrange objects...

Please sign up or login with your details

Forgot password? Click here to reset