Prioritized Level Replay

by   Minqi Jiang, et al.

Simulated environments with procedurally generated content have become popular benchmarks for testing systematic generalization of reinforcement learning agents. Every level in such an environment is algorithmically created, thereby exhibiting a unique configuration of underlying factors of variation, such as layout, positions of entities, asset appearances, or even the rules governing environment transitions. Fixed sets of training levels can be determined to aid comparison and reproducibility, and test levels can be held out to evaluate the generalization and robustness of agents. While prior work samples training levels in a direct way (e.g. uniformly) for the agent to learn from, we investigate the hypothesis that different levels provide different learning progress for an agent at specific times during training. We introduce Prioritized Level Replay, a general framework for estimating the future learning potential of a level given the current state of the agent's policy. We find that temporal-difference (TD) errors, while previously used to selectively sample past transitions, also prove effective for scoring a level's future learning potential in generating entire episodes that an agent would experience when replaying it. We report significantly improved sample-efficiency and generalization on the majority of Procgen Benchmark environments as well as two challenging MiniGrid environments. Lastly, we present a qualitative analysis showing that Prioritized Level Replay induces an implicit curriculum, taking the agent gradually from easier to harder levels.


MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization

Experience replay is crucial for off-policy reinforcement learning (RL) ...

Sample Efficiency in Sparse Reinforcement Learning: Or Your Money Back

Sparse rewards present a difficult problem in reinforcement learning and...

Improving Experience Replay through Modeling of Similar Transitions' Sets

In this work, we propose and evaluate a new reinforcement learning metho...

Quantifying environment and population diversity in multi-agent reinforcement learning

Generalization is a major challenge for multi-agent reinforcement learni...

Leveraging Procedural Generation to Benchmark Reinforcement Learning

In this report, we introduce Procgen Benchmark, a suite of 16 procedural...

Replay-Guided Adversarial Environment Design

Deep reinforcement learning (RL) agents may successfully generalize to n...

Diversity Induced Environment Design via Self-Play

Recent work on designing an appropriate distribution of environments has...

Please sign up or login with your details

Forgot password? Click here to reset