Learning with Good Feature Representations in Bandits and in RL with a Generative Model

by   Tor Lattimore, et al.

The construction in the recent paper by Du et al. [2019] implies that searching for a near-optimal action in a bandit sometimes requires examining essentially all the actions, even if the learner is given linear features in R^d that approximate the rewards with a small uniform error. In this note we use the Kiefer-Wolfowitz theorem to show that by checking only a few actions, a learner can always find an action which is suboptimal with an error of at most O(ε√(d)) where ε is the approximation error of the features. Thus, features are useful when the approximation error is small relative to the dimensionality of the features. The idea is applied to stochastic bandits and reinforcement learning with a generative model where the learner has access to d-dimensional linear features that approximate the action-value functions for all policies to an accuracy of ε. For bandits we prove a bound on the regret of order √(dn log(k)) + ε n √(d)log(n) with k the number of actions and n the horizon. For RL we show that approximate policy iteration can learn a policy that is optimal up to an additive error of order ε√(d) / (1 - γ)^2 and using about d / (ε^2(1-γ)^4) samples from the generative model.


page 1

page 2

page 3

page 4


Does Sparsity Help in Learning Misspecified Linear Bandits?

Recently, the study of linear misspecified bandits has generated intrigu...

Boosting Offline Reinforcement Learning with Residual Generative Modeling

Offline reinforcement learning (RL) tries to learn the near-optimal poli...

Reinforcement Leaning in Feature Space: Matrix Bandit, Kernels, and Regret Bound

Exploration in reinforcement learning (RL) suffers from the curse of dim...

Online Sparse Reinforcement Learning

We investigate the hardness of online reinforcement learning in fixed ho...

Near-optimal Representation Learning for Linear Bandits and Linear RL

This paper studies representation learning for multi-task linear bandits...

No-Regret Linear Bandits beyond Realizability

We study linear bandits when the underlying reward function is not linea...

Linear Bandits with Memory: from Rotting to Rising

Nonstationary phenomena, such as satiation effects in recommendation, ar...

Please sign up or login with your details

Forgot password? Click here to reset