Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processes

by   Jian Qian, et al.
Cole Normale Suprieure

We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses. SCAL^+ is a variant of SCAL (Fruit et al., 2018) that performs efficient exploration-exploitation in any unknown weakly-communicating MDP for which an upper bound C on the span of the optimal bias function is known. For an MDP with S states, A actions and Γ≤ S possible next states, we prove that SCAL^+ achieves the same theoretical guarantees as SCAL (i.e., a high probability regret bound of O(C√(Γ SAT))), with a much smaller computational complexity. Similarly, C-SCAL^+ exploits an exploration bonus to achieve sublinear regret in any undiscounted MDP with continuous state space. We show that C-SCAL^+ achieves the same regret bound as UCCRL (Ortner and Ryabko, 2012) while being the first implementable algorithm with regret guarantees in this setting. While optimistic algorithms such as UCRL, SCAL or UCCRL maintain a high-confidence set of plausible MDPs around the true unknown MDP, SCAL^+ and C-SCAL^+ leverage on an exploration bonus to directly plan on the empirically estimated MDP, thus being more computationally efficient.


page 1

page 2

page 3

page 4


Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes

While designing the state space of an MDP, it is common to include state...

Improved Analysis of UCRL2 with Empirical Bernstein Inequality

We consider the problem of exploration-exploitation in communicating Mar...

Active Exploration in Markov Decision Processes

We introduce the active exploration problem in Markov decision processes...

Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning

We introduce SCAL, an algorithm designed to perform efficient exploratio...

Large Scale Markov Decision Processes with Changing Rewards

We consider Markov Decision Processes (MDPs) where the rewards are unkno...

Causal Markov Decision Processes: Learning Good Interventions Efficiently

We introduce causal Markov Decision Processes (C-MDPs), a new formalism ...

Exploration-Exploitation in Constrained MDPs

In many sequential decision-making problems, the goal is to optimize a u...

Please sign up or login with your details

Forgot password? Click here to reset