Multi-agent Reinforcement Learning with Sparse Interactions by Negotiation and Knowledge Transfer

by   Luowei Zhou, et al.

Reinforcement learning has significant applications for multi-agent systems, especially in unknown dynamic environments. However, most multi-agent reinforcement learning (MARL) algorithms suffer from such problems as exponential computation complexity in the joint state-action space, which makes it difficult to scale up to realistic multi-agent problems. In this paper, a novel algorithm named negotiation-based MARL with sparse interactions (NegoSI) is presented. In contrast to traditional sparse-interaction based MARL algorithms, NegoSI adopts the equilibrium concept and makes it possible for agents to select the non-strict Equilibrium Dominating Strategy Profile (non-strict EDSP) or Meta equilibrium for their joint actions. The presented NegoSI algorithm consists of four parts: the equilibrium-based framework for sparse interactions, the negotiation for the equilibrium set, the minimum variance method for selecting one joint action and the knowledge transfer of local Q-values. In this integrated algorithm, three techniques, i.e., unshared value functions, equilibrium solutions and sparse interactions are adopted to achieve privacy protection, better coordination and lower computational complexity, respectively. To evaluate the performance of the presented NegoSI algorithm, two groups of experiments are carried out regarding three criteria: steps of each episode (SEE), rewards of each episode (REE) and average runtime (AR). The first group of experiments is conducted using six grid world games and shows fast convergence and high scalability of the presented algorithm. Then in the second group of experiments NegoSI is applied to an intelligent warehouse problem and simulated results demonstrate the effectiveness of the presented NegoSI algorithm compared with other state-of-the-art MARL algorithms.


page 1

page 2

page 3

page 4


Multi-agent Reinforcement Learning Improvement in a Dynamic Environment Using Knowledge Transfer

Cooperative multi-agent systems are being widely used in variety of area...

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) is useful in many problems tha...

Pareto Actor-Critic for Equilibrium Selection in Multi-Agent Reinforcement Learning

Equilibrium selection in multi-agent games refers to the problem of sele...

Learning Individual Policies in Large Multi-agent Systems through Local Variance Minimization

In multi-agent systems with large number of agents, typically the contri...

The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning

Recent years have seen the application of deep reinforcement learning te...

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

We consider model-based multi-agent reinforcement learning, where the en...

Channel Selection for Wi-Fi 7 Multi-Link Operation via Optimistic-Weighted VDN and Parallel Transfer Reinforcement Learning

Dense and unplanned IEEE 802.11 Wireless Fidelity(Wi-Fi) deployments and...

Please sign up or login with your details

Forgot password? Click here to reset