Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas

by   Yen-Ling Kuo, et al.

We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. The input LTL formulas have never been seen before, yet the network performs zero-shot generalization to satisfy them. This is a novel form of multi-task learning for RL agents where agents learn from one diverse set of tasks and generalize to a new set of diverse tasks. The formulation of the network enables this capacity to generalize. We demonstrate this ability in two domains. In a symbolic domain, the agent finds a sequence of letters in that are accepted. In a Minecraft-like environment, the agent finds a sequence of actions that conform to the formula. While prior work could learn to execute one formula reliably given examples of that formula, we demonstrate how to encode all formulas reliably. This could form the basis of new multi-task agents that discover sub-tasks and execute them without any additional training, as well as the agents which follow more complex linguistic commands. The structures required for this generalization are specific to LTL formulas, which opens up an interesting theoretical question: what structures are required in neural networks for zero-shot generalization to different logics?


page 2

page 4


Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learning

Humans are capable of abstracting various tasks as different combination...

Programmable Agents

We build deep RL agents that execute declarative programs expressed in f...

Environment Generation for Zero-Shot Compositional Reinforcement Learning

Many real-world problems are compositional - solving them requires compl...

Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning

Assembly of multi-part physical structures is both a valuable end produc...

Multitask Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies

We introduce a new RL problem where the agent is required to execute a g...

Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations

There has been great progress in unifying various table-to-text tasks us...

Compositional Multi-Object Reinforcement Learning with Linear Relation Networks

Although reinforcement learning has seen remarkable progress over the la...

Please sign up or login with your details

Forgot password? Click here to reset