Behavior Contrastive Learning for Unsupervised Skill Discovery

by   Rushuai Yang, et al.

In reinforcement learning, unsupervised skill discovery aims to learn diverse skills without extrinsic rewards. Previous methods discover skills by maximizing the mutual information (MI) between states and skills. However, such an MI objective tends to learn simple and static skills and may hinder exploration. In this paper, we propose a novel unsupervised skill discovery method through contrastive learning among behaviors, which makes the agent produce similar behaviors for the same skill and diverse behaviors for different skills. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill, which serves as an upper bound of the previous MI objective. Meanwhile, our method implicitly increases the state entropy to obtain better state coverage. We evaluate our method on challenging mazes and continuous control tasks. The results show that our method generates diverse and far-reaching skills, and also obtains competitive performance in downstream tasks compared to the state-of-the-art methods.


page 4

page 7

page 9

page 19

page 20

page 21


CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsup...

Unsupervised Discovery of Continuous Skills on a Sphere

Recently, methods for learning diverse skills to generate various behavi...

Lipschitz-constrained Unsupervised Skill Discovery

We study the problem of unsupervised skill discovery, whose goal is to l...

Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching

Learning meaningful behaviors in the absence of reward is a difficult pr...

DisTop: Discovering a Topological representation to learn diverse and rewarding skills

The optimal way for a deep reinforcement learning (DRL) agent to explore...

Choreographer: Learning and Adapting Skills in Imagination

Unsupervised skill learning aims to learn a rich repertoire of behaviors...

Diverse Offline Imitation via Fenchel Duality

There has been significant recent progress in the area of unsupervised s...

Please sign up or login with your details

Forgot password? Click here to reset