First-Explore, then Exploit: Meta-Learning Intelligent Exploration

by   Ben Norman, et al.

Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. by taking into account complex domain priors and previous explorations). Even the most basic intelligent exploration strategies such as exhaustive search are only inefficiently or poorly approximated by approaches such as novelty search or intrinsic motivation, let alone more complicated strategies like learning new skills, climbing stairs, opening doors, or conducting experiments. This lack of intelligent exploration limits sample efficiency and prevents solving hard exploration domains. We argue a core barrier prohibiting many RL approaches from learning intelligent exploration is that the methods attempt to explore and exploit simultaneously, which harms both exploration and exploitation as the goals often conflict. We propose a novel meta-RL framework (First-Explore) with two policies: one policy learns to only explore and one policy learns to only exploit. Once trained, we can then explore with the explore policy, for as long as desired, and then exploit based on all the information gained during exploration. This approach avoids the conflict of trying to do both exploration and exploitation at once. We demonstrate that First-Explore can learn intelligent exploration strategies such as exhaustive search and more, and that it outperforms dominant standard RL and meta-RL approaches on domains where exploration requires sacrificing reward. First-Explore is a significant step towards creating meta-RL algorithms capable of learning human-level exploration which is essential to solve challenging unseen hard-exploration domains.


page 8

page 16

page 18


Show me the Way: Intrinsic Motivation from Demonstrations

The study of exploration in Reinforcement Learning (RL) has a long histo...

Learn to Effectively Explore in Context-Based Meta-RL

Meta reinforcement learning (meta-RL) provides a principled approach for...

Learning Context-aware Task Reasoning for Efficient Meta-reinforcement Learning

Despite recent success of deep network-based Reinforcement Learning (RL)...

earning Context-aware Task Reasoning for Efficient Meta Reinforcement Learning

Despite recent success of deep network-based Reinforcement Learning (RL)...

Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network

Existing exploration strategies in reinforcement learning (RL) often eit...

Polyethism in a colony of artificial ants

We explore self-organizing strategies for role assignment in a foraging ...

Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning

We seek to efficiently learn by leveraging shared structure between diff...

Please sign up or login with your details

Forgot password? Click here to reset