Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

01/31/2022
by   Denis Yarats, et al.
10

Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because offline RL data is usually collected to optimize specific target tasks limiting the data's diversity. In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks. Our findings suggest that data generation is as important as algorithmic advances for offline RL and hence requires careful consideration from the community.

READ FULL TEXT

page 2

page 4

page 5

page 6

page 8

page 16

page 17

page 18

02/11/2021

Representation Matters: Offline Pretraining for Sequential Decision Making

The recent success of supervised learning methods on ever larger offline...
02/01/2021

NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims at learning a good policy from ...
02/27/2023

The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning

Self-supervised methods have become crucial for advancing deep learning ...
10/12/2022

Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories

Natural agents can effectively learn from multiple data sources that dif...
12/07/2020

Efficient Reservoir Management through Deep Reinforcement Learning

Dams impact downstream river dynamics through flow regulation and disrup...
06/01/2023

Improving and Benchmarking Offline Reinforcement Learning Algorithms

Recently, Offline Reinforcement Learning (RL) has achieved remarkable pr...
05/18/2023

Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models

Recently, reward-conditioned reinforcement learning (RCRL) has gained po...

Please sign up or login with your details

Forgot password? Click here to reset