Actor-Critic Method for High Dimensional Static Hamilton–Jacobi–Bellman Partial Differential Equations based on Neural Networks

02/22/2021

∙

We propose a novel numerical method for high dimensional Hamilton–Jacobi–Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least square temporal difference method (VR-LSTD) using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive stepsize scheme to improve the accuracy near the domain boundary. Numerical examples up to 20 spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, and the diffusive Eikonal equations are presented to validate the effectiveness of our proposed method.

READ FULL TEXT

Actor-Critic Method for High Dimensional Static Hamilton–Jacobi–Bellman Partial Differential Equations based on Neural Networks

Actor-Critic Algorithm for High-dimensional Partial Differential Equations

Actor-Critic learning for mean-field control in continuous time

On Parametric Optimal Execution and Machine Learning Surrogates

Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packs

General solutions for nonlinear differential equations: a deep reinforcement learning approach

Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning

Adaptive Bases for Reinforcement Learning

Actor-Critic Method for High Dimensional Static Hamilton–Jacobi–Bellman Partial Differential Equations based on Neural Networks

Related Research

Actor-Critic Algorithm for High-dimensional Partial Differential Equations

Actor-Critic learning for mean-field control in continuous time

On Parametric Optimal Execution and Machine Learning Surrogates

Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packs

General solutions for nonlinear differential equations: a deep reinforcement learning approach

Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning

Adaptive Bases for Reinforcement Learning