Overoptimization Failures and Specification Gaming in Multi-agent Systems

10/16/2018

∙

by   David Manheim, et al.

∙

0

∙

share

Overoptimization failures in machine learning and AI can involve specification gaming, reward hacking, fragility to distributional shifts, and Goodhart's or Campbell's law. These failure modes are an important challenge in building safe AI systems, but multi-agent systems have additional related failure modes. These failure modes are more complex, more problematic, and less well understood in the multi-agent setting, at least partially because they are not yet observed in practice. This paper explains why this is the case, then lays out some of the classes of such failure, such as accidental steering, coordination failures, adversarial misalignment, input spoofing or filtering, and goal co-option or direct hacking.