Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion

by   Siddhant Gangapurwala, et al.
University of Oxford

Deep reinforcement learning (RL) uses model-free techniques to optimize task-specific control policies. Despite having emerged as a promising approach for complex problems, RL is still hard to use reliably for real-world applications. Apart from challenges such as precise reward function tuning, inaccurate sensing and actuation, and non-deterministic response, existing RL methods do not guarantee behavior within required safety constraints that are crucial for real robot scenarios. In this regard, we introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained proximal policy optimization (CPPO) for tracking base velocity commands while following the defined constraints. We also introduce schemes which encourage state recovery into constrained regions in case of constraint violations. We present experimental results of our training method and test it on the real ANYmal quadruped robot. We compare our approach against the unconstrained RL method and show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.


page 1

page 4

page 7


Inverse Constrained Reinforcement Learning

Standard reinforcement learning (RL) algorithms train agents to maximize...

Reinforcement Learning Agent Training with Goals for Real World Tasks

Reinforcement Learning (RL) is a promising approach for solving various ...

Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety

The safety constraints commonly used by existing safe reinforcement lear...

Value constrained model-free continuous control

The naive application of Reinforcement Learning algorithms to continuous...

Advanced Skills through Multiple Adversarial Motion Priors in Reinforcement Learning

In recent years, reinforcement learning (RL) has shown outstanding perfo...

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Accurate and precise terrain estimation is a difficult problem for robot...

Projection-Based Constrained Policy Optimization

We consider the problem of learning control policies that optimize a rew...

Please sign up or login with your details

Forgot password? Click here to reset