Learning Parametric Closed-Loop Policies for Markov Potential Games

by   Sergio Valcarcel Macua, et al.

Multiagent systems where the agents interact among themselves and with an stochastic environment can be formalized as stochastic games. We study a subclass, named Markov potential games (MPGs), that appear often in economic and engineering applications when the agents share some common resource. We consider MPGs with continuous state-action variables, coupled constraints and nonconvex rewards. Previous analysis are only valid for very simple cases (convex rewards, invertible dynamics, and no coupled constraints); or considered deterministic dynamics and provided open-loop (OL) analysis, studying strategies that consist in predefined action sequences. We present a closed-loop (CL) analysis for MPGs and consider parametric policies that depend on the current state and where agents adapt to stochastic transitions. We provide verifiable, sufficient and necessary conditions for a stochastic game to be an MPG, even for complex parametric functions (e.g., deep neural networks); and show that a CL Nash equilibrium (NE) can be found (or at least approximated) by solving a related optimal control problem (OCP). This is useful since solving an OCP---a single-objective problem---is usually much simpler than solving the original set of coupled OCPs that form the game---a multiobjective control problem. This is a considerable improvement over previously standard approach. We illustrate the theoretical contributions with an example by applying our approach to a noncooperative communications engineering game. We then solve the game with a deep reinforcement learning algorithm that learns policies that closely approximates an exact variational NE of the game.


page 1

page 2

page 3

page 4


Markov Games with Decoupled Dynamics: Price of Anarchy and Sample Complexity

This paper studies the finite-time horizon Markov games where the agents...

Efficient Constrained Multi-Agent Interactive Planning using Constrained Dynamic Potential Games

Although dynamic games provide a rich paradigm for modeling agents' inte...

Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions

In this paper, we study the problem of multiple stochastic agents intera...

Decentralized Policy Gradient for Nash Equilibria Learning of General-sum Stochastic Games

We study Nash equilibria learning of a general-sum stochastic game with ...

Decentralised Learning in Systems with Many, Many Strategic Agents

Although multi-agent reinforcement learning can tackle systems of strate...

A distributed algorithm for average aggregative games with coupling constraints

We consider the framework of average aggregative games, where the cost f...

Optimal strategies in the Fighting Fantasy gaming system: influencing stochastic dynamics by gambling with limited resource

Fighting Fantasy is a popular recreational fantasy gaming system worldwi...

Please sign up or login with your details

Forgot password? Click here to reset