Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies

by   Tsung-Yen Yang, et al.

We consider the problem of reinforcement learning when provided with a baseline control policy and a set of constraints that the controlled system must satisfy. The baseline policy might arise from a heuristic, a prior application, a teacher or demonstrator data. The constraints might encode safety, fairness or some application-specific requirements. We want to efficiently use reinforcement learning to adapt the baseline policy to improve performance and satisfy the given constraints when it is applied to the new system. The key challenge is to effectively use the baseline policy (which need not satisfy the current constraints) to aid the learning of a constraint-satisfying policy in the new application. We propose an iterative algorithm for solving this problem. Each iteration is composed of three-steps. The first step performs a policy update to increase the expected reward, the second step performs a projection to minimize the distance between the current policy and the baseline policy, and the last step performs a projection onto the set of policies that satisfy the constraints. This procedure allows the learning process to leverage the baseline policy to achieve faster learning while improving reward performance and satisfying the constraints imposed on the current problem. We analyze the convergence of the proposed algorithm and provide a finite-sample guarantee. Empirical results demonstrate that the algorithm can achieve superior performance, with 10 times fewer constraint violations and around 40


page 6

page 29

page 31


Projection-Based Constrained Policy Optimization

We consider the problem of learning control policies that optimize a rew...

Set-Invariant Constrained Reinforcement Learning with a Meta-Optimizer

This paper investigates reinforcement learning with safety constraints. ...

Batch Policy Learning under Constraints

When learning policies for real-world domains, two important questions a...

Lyapunov Barrier Policy Optimization

Deploying Reinforcement Learning (RL) agents in the real-world require t...

Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks

This paper focuses on finding reinforcement learning policies for contro...

Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization

While reinforcement learning (RL) is gaining popularity in energy system...

Solving Constrained Reinforcement Learning through Augmented State and Reward Penalties

Constrained Reinforcement Learning has been employed to enforce safety c...

Please sign up or login with your details

Forgot password? Click here to reset