Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

by   Timothy Verstraeten, et al.

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible. In this work, we focus on learning to coordinate. Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We provide a regret bound that is sublinear in time and low-order polynomial in the highest number of actions of a single agent for sparse coordination graphs. Additionally, we empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks, and a novel benchmark with Poisson distributions. An example of a loosely-coupled multi-agent system is a wind farm. Coordination within the wind farm is necessary to maximize power production. As upstream wind turbines only affect nearby downstream turbines, we can use MATS to efficiently learn the optimal control mechanism for the farm. To demonstrate the benefits of our method toward applications we apply MATS to a realistic wind farm control task. In this task, wind turbines must coordinate their alignments with respect to the incoming wind vector in order to optimize power production. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in practical applications with sparse neighbourhood structures.


Thompson Sampling for Factored Multi-Agent Bandits

Multi-agent coordination is prevalent in many real-world applications. H...

Scalable Optimization for Wind Farm Control using Coordination Graphs

Wind farms are a crucial driver toward the generation of ecological and ...

Improving Multi-agent Coordination by Learning to Estimate Contention

We present a multi-agent learning algorithm, ALMA-Learning, for efficien...

A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

Training multiple agents to coordinate is an important problem with appl...

Design of a GIS-based Assistant Software Agent for the Incident Commander to Coordinate Emergency Response Operations

Problem: This paper addresses the design of an intelligent software syst...

Robust Semi-Cooperative Multi-Agent Coordination in the Presence of Stochastic Disturbances

This paper presents a robust distributed coordination protocol that achi...

Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Models

Many real-world problems contain multiple objectives and agents, where a...

Please sign up or login with your details

Forgot password? Click here to reset