DeepAI AI Chat
Log In Sign Up

Towards Data-Driven Offline Simulations for Online Reinforcement Learning

by   Shengpu Tang, et al.

Modern decision-making systems, from robots to web recommendation engines, are expected to adapt: to user preferences, changing circumstances or even new tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather than a fixed policy) to a production system, as it's perceived as unsafe. Using historical data to reason about learning algorithms, similar to offline policy evaluation (OPE) applied to fixed policies, could help practitioners evaluate and ultimately deploy such adaptive agents to production. In this work, we formalize offline learner simulation (OLS) for reinforcement learning (RL) and propose a novel evaluation protocol that measures both fidelity and efficiency of the simulation. For environments with complex high-dimensional observations, we propose a semi-parametric approach that leverages recent advances in latent state discovery in order to achieve accurate and efficient offline simulations. In preliminary experiments, we show the advantage of our approach compared to fully non-parametric baselines. The code to reproduce these experiments will be made available at


page 12

page 13


Offline Reinforcement Learning Hands-On

Offline Reinforcement Learning (RL) aims to turn large datasets into pow...

Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning

We study offline meta-reinforcement learning, a practical reinforcement ...

Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation

Powered by deep representation learning, reinforcement learning (RL) pro...

Personalization for Web-based Services using Offline Reinforcement Learning

Large-scale Web-based services present opportunities for improving UI po...

Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation

Most of the existing deep reinforcement learning (RL) approaches for ses...

DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs

We study an approach to offline reinforcement learning (RL) based on opt...