Safety-Aware Task Composition for Discrete and Continuous Reinforcement Learning

by   Kevin Leahy, et al.

Compositionality is a critical aspect of scalable system design. Reinforcement learning (RL) has recently shown substantial success in task learning, but has only recently begun to truly leverage composition. In this paper, we focus on Boolean composition of learned tasks as opposed to functional or sequential composition. Existing Boolean composition for RL focuses on reaching a satisfying absorbing state in environments with discrete action spaces, but does not support composable safety (i.e., avoidance) constraints. We advance the state of the art in Boolean composition of learned tasks with three contributions: i) introduce two distinct notions of safety in this framework; ii) show how to enforce either safety semantics, prove correctness (under some assumptions), and analyze the trade-offs between the two safety notions; and iii) extend Boolean composition from discrete action spaces to continuous action spaces. We demonstrate these techniques using modified versions of value iteration in a grid world, Deep Q-Network (DQN) in a grid world with image observations, and Twin Delayed DDPG (TD3) in a continuous-observation and continuous-action Bullet physics environment. We believe that these contributions advance the theory of safe reinforcement learning by allowing zero-shot composition of policies satisfying safety properties.


page 9

page 17

page 18


Safe Exploration in Continuous Action Spaces

We address the problem of deploying a reinforcement learning (RL) agent ...

Will it Blend? Composing Value Functions in Reinforcement Learning

An important property for lifelong-learning agents is the ability to com...

State-wise Safe Reinforcement Learning: A Survey

Despite the tremendous success of Reinforcement Learning (RL) algorithms...

Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Models

Safety is one of the biggest concerns to applying reinforcement learning...

Bounding the Optimal Value Function in Compositional Reinforcement Learning

In the field of reinforcement learning (RL), agents are often tasked wit...

Entropic Policy Composition with Generalized Policy Improvement and Divergence Correction

Deep reinforcement learning (RL) algorithms have made great strides in r...

Please sign up or login with your details

Forgot password? Click here to reset