Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees

09/28/2022
by   Daniil Tiapkin, et al.
0

We consider reinforcement learning in an environment modeled by an episodic, finite, stage-dependent Markov decision process of horizon H with S states, and A actions. The performance of an agent is measured by the regret after interacting with the environment for T episodes. We propose an optimistic posterior sampling algorithm for reinforcement learning (OPSRL), a simple variant of posterior sampling that only needs a number of posterior samples logarithmic in H, S, A, and T per state-action pair. For OPSRL we guarantee a high-probability regret bound of order at most 𝒪(√(H^3SAT)) ignoring polylog(HSAT) terms. The key novel technical ingredient is a new sharp anti-concentration inequality for linear forms which may be of independent interest. Specifically, we extend the normal approximation-based lower bound for Beta distributions by Alfers and Dinges [1984] to Dirichlet distributions. Our bound matches the lower bound of order Ω(√(H^3SAT)), thereby answering the open problems raised by Agrawal and Jia [2017b] for the episodic setting.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset