DeepAI AI Chat
Log In Sign Up

Influencing Long-Term Behavior in Multiagent Reinforcement Learning

by   Dong-Ki Kim, et al.
University of Oxford

The main challenge of multiagent reinforcement learning is the difficulty of learning useful policies in the presence of other simultaneously learning agents whose changing behaviors jointly affect the environment's transition and reward dynamics. An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit. Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only a finite number of policy updates. As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence. In this paper, we propose a principled framework for considering the limiting policies of other agents as time approaches infinity. Specifically, we develop a new optimization objective that maximizes each agent's average reward by directly accounting for the impact of its behavior on the limiting set of policies that other agents will converge to. Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing over possible outcomes. As a result of our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across a suite of diverse multiagent benchmark domains.


page 1

page 2

page 3

page 4


Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibria

Multiagent learning settings are inherently more difficult than single-a...

Influence-based Reinforcement Learning for Intrinsically-motivated Agents

The reinforcement learning (RL) research area is very active, with sever...

On-Policy Deep Reinforcement Learning for the Average-Reward Criterion

We develop theory and algorithms for average-reward on-policy Reinforcem...

PAC Reinforcement Learning with Rich Observations

We propose and study a new model for reinforcement learning with rich ob...

ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation

In a multirobot system, a number of cyber-physical attacks (e.g., commun...

Optimal Aggregation Strategies for Social Learning over Graphs

Adaptive social learning is a useful tool for studying distributed decis...