Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

10/25/2022
∙
by   Hao Liang, et al.
∙
0
∙

We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective is the entropic risk measure (EntRM) of return. We identify a key property of the EntRM, the monotonicity-preserving property, which enables the risk-sensitive distributional dynamic programming framework. We then propose two novel DRL algorithms that implement optimism through two different schemes, including a model-free one and a model-based one. We prove that both of them attain 𝒪̃(exp(|β| H)-1/|β|HH√(HS^2AT)) regret upper bound, where S is the number of states, A the number of states, H the time horizon and T the number of total time steps. It matches RSVI2 proposed in <cit.> with a much simpler regret analysis. To the best of our knowledge, this is the first regret analysis of DRL, which bridges DRL and RSRL in terms of sample complexity. Finally, we improve the existing lower bound by proving a tighter bound of Ω(exp(β H/6)-1/β HH√(SAT)) for β>0 case, which recovers the tight lower bound Ω(H√(SAT)) in the risk-neutral setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
∙ 06/22/2020

Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret

We study risk-sensitive reinforcement learning in episodic Markov decisi...
research
∙ 10/20/2022

Horizon-Free Reinforcement Learning for Latent Markov Decision Processes

We study regret minimization for reinforcement learning (RL) in Latent M...
research
∙ 07/19/2019

Delegative Reinforcement Learning: learning to avoid traps with a little help

Most known regret bounds for reinforcement learning are either episodic ...
research
∙ 07/04/2023

Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning

We consider the problem of learning models for risk-sensitive reinforcem...
research
∙ 03/07/2022

Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning

In this paper, we study gap-dependent regret guarantees for risk-sensiti...
research
∙ 02/25/2023

Exponential Hardness of Reinforcement Learning with Linear Function Approximation

A fundamental question in reinforcement learning theory is: suppose the ...
research
∙ 11/06/2021

Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning

We study risk-sensitive reinforcement learning (RL) based on the entropi...

Please sign up or login with your details

Forgot password? Click here to reset