Infinite-Horizon Reach-Avoid Zero-Sum Games via Deep Reinforcement Learning

by   Jingqi Li, et al.

In this paper, we consider the infinite-horizon reach-avoid zero-sum game problem, where the goal is to find a set in the state space, referred to as the reach-avoid set, such that the system starting at a state therein could be controlled to reach a given target set without violating constraints under the worst-case disturbance. We address this problem by designing a new value function with a contracting Bellman backup, where the super-zero level set, i.e., the set of states where the value function is evaluated to be non-negative, recovers the reach-avoid set. Building upon this, we prove that the proposed method can be adapted to compute the viability kernel, or the set of states which could be controlled to satisfy given constraints, and the backward reachable set, or the set of states that could be driven towards a given target set. Finally, we propose to alleviate the curse of dimensionality issue in high-dimensional problems by extending Conservative Q-Learning, a deep reinforcement learning technique, to learn a value function such that the super-zero level set of the learned value function serves as a (conservative) approximation to the reach-avoid set. Our theoretical and empirical results suggest that the proposed method could learn reliably the reach-avoid set and the optimal control policy even with neural network approximation.


page 2

page 6


Value Function Approximation in Zero-Sum Markov Games

This paper investigates value function approximation in the context of z...

Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning

Reach-avoid optimal control problems, in which the system must reach cer...

Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems

Numerical methods for the optimal feedback control of high-dimensional d...

Real-Time Optimal Guidance and Control for Interplanetary Transfers Using Deep Networks

We consider the Earth-Venus mass-optimal interplanetary transfer of a lo...

Reach-Avoid Problems via Sum-of-Squares Optimization and Dynamic Programming

Reach-avoid problems involve driving a system to a set of desirable conf...

Generation of Policy-Level Explanations for Reinforcement Learning

Though reinforcement learning has greatly benefited from the incorporati...

State Space Closure: Revisiting Endless Online Level Generation via Reinforcement Learning

In this paper we revisit endless online level generation with the recent...

Please sign up or login with your details

Forgot password? Click here to reset