Reconnaissance and Planning algorithm for constrained MDP

by   Shin-ichi Maeda, et al.
Preferred Infrastructure
The University of Tokyo

Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we propose a novel simulator-based method to approximately solve a CMDP problem without making any compromise on the safety constraints. We achieve this by decomposing the CMDP into a pair of MDPs; reconnaissance MDP and planning MDP. The purpose of reconnaissance MDP is to evaluate the set of actions that are safe, and the purpose of planning MDP is to maximize the return while using the actions authorized by reconnaissance MDP. RMDP can define a set of safe policies for any given set of safety constraint, and this set of safe policies can be used to solve another CMDP problem with different reward. Our method is not only computationally less demanding than the previous simulator-based approaches to CMDP, but also capable of finding a competitive reward-seeking policy in a high dimensional environment, including those involving multiple moving obstacles.


page 3

page 4

page 5

page 6

page 11

page 12

page 13

page 14


Safe Reinforcement Learning Using Advantage-Based Intervention

Many sequential decision problems involve finding a policy that maximize...

Reinforcement Learning for Task Specifications with Action-Constraints

In this paper, we use concepts from supervisory control theory of discre...

Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects

In safe MDP planning, a cost function based on the current state and act...

Safe Exploration by Solving Early Terminated MDP

Safe exploration is crucial for the real-world application of reinforcem...

Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation

We address the problem of safe reinforcement learning from pixel observa...

Hierarchical Motion Planning under Probabilistic Temporal Tasks and Safe-Return Constraints

Safety is crucial for robotic missions within an uncertain environment. ...

Formal Language Constraints for Markov Decision Processes

In order to satisfy safety conditions, a reinforcement learned (RL) agen...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset