On-the-fly Strategy Adaptation for ad-hoc Agent Coordination

by   Jaleh Zand, et al.

Training agents in cooperative settings offers the promise of AI agents able to interact effectively with humans (and other agents) in the real world. Multi-agent reinforcement learning (MARL) has the potential to achieve this goal, demonstrating success in a series of challenging problems. However, whilst these advances are significant, the vast majority of focus has been on the self-play paradigm. This often results in a coordination problem, caused by agents learning to make use of arbitrary conventions when playing with themselves. This means that even the strongest self-play agents may have very low cross-play with other agents, including other initializations of the same algorithm. In this paper we propose to solve this problem by adapting agent strategies on the fly, using a posterior belief over the other agents' strategy. Concretely, we consider the problem of selecting a strategy from a finite set of previously trained agents, to play with an unknown partner. We propose an extension of the classic statistical technique, Gibbs sampling, to update beliefs about other agents and obtain close to optimal ad-hoc performance. Despite its simplicity, our method is able to achieve strong cross-play with unseen partners in the challenging card game of Hanabi, achieving successful ad-hoc coordination without knowledge of the partner's strategy a priori.


page 1

page 2

page 3

page 4


Evaluating the Rainbow DQN Agent in Hanabi with Unseen Partners

Hanabi is a cooperative game that challenges exist-ing AI techniques due...

Modeling Human Ad Hoc Coordination

Whether in groups of humans or groups of computer agents, collaboration ...

Generalized Beliefs for Cooperative AI

Self-play is a common paradigm for constructing solutions in Markov game...

Complexity Reduction in the Negotiation of New Lexical Conventions

In the process of collectively inventing new words for new con- cepts in...

M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

Most of the prior work on multi-agent reinforcement learning (MARL) achi...

Learning to Cooperate with Unseen Agent via Meta-Reinforcement Learning

Ad hoc teamwork problem describes situations where an agent has to coope...

Learning Best Response Strategies for Agents in Ad Exchanges

Ad exchanges are widely used in platforms for online display advertising...

Please sign up or login with your details

Forgot password? Click here to reset