A Game Theoretic Framework for Model Based Reinforcement Learning

by   Aravind Rajeswaran, et al.

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation.


page 8

page 26

page 27


Model-Based Reinforcement Learning via Meta-Policy Optimization

Model-based reinforcement learning approaches carry the promise of being...

A Sharp Analysis of Model-based Reinforcement Learning with Self-Play

Model-based algorithms—algorithms that decouple learning of the model an...

Uncertainty-aware Model-based Policy Optimization

Model-based reinforcement learning has the potential to be more sample e...

When to Trust Your Model: Model-Based Policy Optimization

Designing effective model-based reinforcement learning algorithms is dif...

Domain Knowledge Integration By Gradient Matching For Sample-Efficient Reinforcement Learning

Model-free deep reinforcement learning (RL) agents can learn an effectiv...

Enhanced Rolling Horizon Evolution Algorithm with Opponent Model Learning: Results for the Fighting Game AI Competition

The Fighting Game AI Competition (FTGAIC) provides a challenging benchma...

An Evolutionary Game Model for Understanding Fraud in Consumption Taxes

This paper presents a computational evolutionary game model to study and...

Please sign up or login with your details

Forgot password? Click here to reset