Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

by   Pier Giuseppe Sessa, et al.

We consider model-based multi-agent reinforcement learning, where the environment transition model is unknown and can only be learned via expensive interactions with the environment. We propose H-MARL (Hallucinated Multi-Agent Reinforcement Learning), a novel sample-efficient algorithm that can efficiently balance exploration, i.e., learning about the environment, and exploitation, i.e., achieve good equilibrium performance in the underlying general-sum Markov game. H-MARL builds high-probability confidence intervals around the unknown transition model and sequentially updates them based on newly observed data. Using these, it constructs an optimistic hallucinated game for the agents for which equilibrium policies are computed at each round. We consider general statistical models (e.g., Gaussian processes, deep ensembles, etc.) and policy classes (e.g., deep neural networks), and theoretically analyze our approach by bounding the agents' dynamic regret. Moreover, we provide a convergence rate to the equilibria of the underlying Markov game. We demonstrate our approach experimentally on an autonomous driving simulation benchmark. H-MARL learns successful equilibrium policies after a few interactions with the environment and can significantly improve the performance compared to non-exploratory methods.


page 7

page 18


Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning

Learning in multi-agent systems is highly challenging due to the inheren...

Provably Efficient Reinforcement Learning in Decentralized General-Sum Markov Games

This paper addresses the problem of learning an equilibrium efficiently ...

Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweeping

We present a new model-based reinforcement learning algorithm, Cooperati...

How Bad is Selfish Driving? Bounding the Inefficiency of Equilibria in Urban Driving Games

We consider the interaction among agents engaging in a driving task and ...

Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning via Best Response

This paper introduces two metrics (cycle-based and memory-based metrics)...

Multi-agent Reinforcement Learning with Sparse Interactions by Negotiation and Knowledge Transfer

Reinforcement learning has significant applications for multi-agent syst...

Please sign up or login with your details

Forgot password? Click here to reset