LISA: Learning Interpretable Skill Abstractions from Language

by   Divyansh Garg, et al.

Learning policies that effectually utilize language instructions in complex, multi-task environments is an important problem in imitation learning. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. To encode complex instructions into skills that can generalize to unseen instructions, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable skills from language-conditioned demonstrations. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA is able to outperform a strong non-hierarchical baseline in the low data regime and compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.


page 7

page 12

page 13

page 14

page 15

page 16

page 20

page 22


Divide Conquer Imitation Learning

When cast into the Deep Reinforcement Learning framework, many robotics ...

SKILL-IL: Disentangling Skill and Knowledge in Multitask Imitation Learning

In this work, we introduce a new perspective for learning transferable c...

Guiding Policies with Language via Meta-Learning

Behavioral skills or policies for autonomous agents are conventionally l...

Text2Motion: From Natural Language Instructions to Feasible Plans

We propose Text2Motion, a language-based planning framework enabling rob...

Complex Skill Acquisition through Simple Skill Adversarial Imitation Learning

Humans are able to think of complex tasks as combinations of simpler sub...

Hierarchical Few-Shot Imitation with Skill Transition Models

A desirable property of autonomous agents is the ability to both solve l...

BridgeData V2: A Dataset for Robot Learning at Scale

We introduce BridgeData V2, a large and diverse dataset of robotic manip...

Please sign up or login with your details

Forgot password? Click here to reset