Multi-Task Off-Policy Learning from Bandit Feedback

by   Joey Hong, et al.

Many practical applications, such as recommender systems and learning to rank, involve solving multiple similar tasks. One example is learning of recommendation policies for users with similar movie preferences, where the users may still rank the individual movies slightly differently. Such tasks can be organized in a hierarchy, where similar tasks are related through a shared structure. In this work, we formulate this problem as a contextual off-policy optimization in a hierarchical graphical model from logged bandit feedback. To solve the problem, we propose a hierarchical off-policy optimization algorithm (HierOPO), which estimates the parameters of the hierarchical model and then acts pessimistically with respect to them. We instantiate HierOPO in linear Gaussian models, for which we also provide an efficient implementation and analysis. We prove per-task bounds on the suboptimality of the learned policies, which show a clear improvement over not using the hierarchical model. We also evaluate the policies empirically. Our theoretical and empirical results show a clear advantage of using the hierarchy over solving each task independently.


page 1

page 2

page 3

page 4


Multi-task Representation Learning with Stochastic Linear Bandits

We study the problem of transfer-learning in the setting of stochastic l...

Hierarchical Bayesian Bandits

Meta-, multi-task, and federated learning can be all viewed as solving s...

Multi-Task Policy Search

Learning policies that generalize across multiple tasks is an important ...

Optimizing Ranking Systems Online as Bandits

Ranking system is the core part of modern retrieval and recommender syst...

Task Selection Policies for Multitask Learning

One of the questions that arises when designing models that learn to sol...

Differentiable Meta-Learning in Contextual Bandits

We study a contextual bandit setting where the learning agent has access...

Hierarchical Exploration for Accelerating Contextual Bandits

Contextual bandit learning is an increasingly popular approach to optimi...

Please sign up or login with your details

Forgot password? Click here to reset