Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior

by   Kai Cui, et al.

Recent reinforcement learning (RL) methods have achieved success in various domains. However, multi-agent RL (MARL) remains a challenge in terms of decentralization, partial observability and scalability to many agents. Meanwhile, collective behavior requires resolution of the aforementioned challenges, and remains of importance to many state-of-the-art applications such as active matter physics, self-organizing systems, opinion dynamics, and biological or robotic swarms. Here, MARL via mean field control (MFC) offers a potential solution to scalability, but fails to consider decentralized and partially observable systems. In this paper, we enable decentralized behavior of agents under partial information by proposing novel models for decentralized partially observable MFC (Dec-POMFC), a broad class of problems with permutation-invariant agents allowing for reduction to tractable single-agent Markov decision processes (MDP) with single-agent RL solution. We provide rigorous theoretical results, including a dynamic programming principle, together with optimality guarantees for Dec-POMFC solutions applied to finite swarms of interest. Algorithmically, we propose Dec-POMFC-based policy gradient methods for MARL via centralized training and decentralized execution, together with policy gradient approximation guarantees. In addition, we improve upon state-of-the-art histogram-based MFC by kernel methods, which is of separate interest also for fully observable MFC. We evaluate numerically on representative collective behavior tasks such as adapted Kuramoto and Vicsek swarming models, being on par with state-of-the-art MARL. Overall, our framework takes a step towards RL-based engineering of artificial collective behavior via MFC.


page 9

page 15

page 18

page 19

page 20


Scalable Task-Driven Robotic Swarm Control via Collision Avoidance and Learning Mean-Field Control

In recent years, reinforcement learning and its multi-agent analogue hav...

Inverse Reinforcement Learning in Swarm Systems

Inverse reinforcement learning (IRL) has become a useful tool for learni...

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

Traditional multi-agent reinforcement learning algorithms are difficultl...

Multi-Agent Learning of Numerical Methods for Hyperbolic PDEs with Factored Dec-MDP

Factored decentralized Markov decision process (Dec-MDP) is a framework ...

Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

We introduce hybrid execution in multi-agent reinforcement learning (MAR...

Hierarchical Reinforcement Learning under Mixed Observability

The framework of mixed observable Markov decision processes (MOMDP) mode...

Using synchronous Boolean networks to model several phenomena of collective behavior

In this paper, we propose an approach for modeling and analysis of a num...

Please sign up or login with your details

Forgot password? Click here to reset