Interactively Learning Preference Constraints in Linear Bandits

by   David Lindner, et al.

We study sequential decision-making with known rewards and unknown constraints, motivated by situations where the constraints represent expensive-to-evaluate human preferences, such as safe and comfortable driving behavior. We formalize the challenge of interactively learning about these constraints as a novel linear bandit problem which we call constrained linear best-arm identification. To solve this problem, we propose the Adaptive Constraint Learning (ACOL) algorithm. We provide an instance-dependent lower bound for constrained linear best-arm identification and show that ACOL's sample complexity matches the lower bound in the worst-case. In the average case, ACOL's sample complexity bound is still significantly tighter than bounds of simpler approaches. In synthetic experiments, ACOL performs on par with an oracle solution and outperforms a range of baselines. As an application, we consider learning constraints to represent human preferences in a driving simulation. ACOL is significantly more sample efficient than alternatives for this application. Further, we find that learning preferences as constraints is more robust to changes in the driving scenario than encoding the preferences directly in the reward function.


Nearly Instance Optimal Sample Complexity Bounds for Top-k Arm Selection

In the Best-k-Arm problem, we are given n stochastic bandit arms, each a...

Best Arm Identification with Fairness Constraints on Subpopulations

We formulate, analyze and solve the problem of best arm identification w...

An Empirical Process Approach to the Union Bound: Practical Algorithms for Combinatorial and Linear Bandits

This paper proposes near-optimal algorithms for the pure-exploration lin...

Active Learning with Safety Constraints

Active learning methods have shown great promise in reducing the number ...

Fully adaptive algorithm for pure exploration in linear bandits

We propose the first fully-adaptive algorithm for pure exploration in li...

Optimal Best-Arm Identification in Bandits with Access to Offline Data

Learning paradigms based purely on offline data as well as those based s...

Nested Elimination: A Simple Algorithm for Best-Item Identification from Choice-Based Feedback

We study the problem of best-item identification from choice-based feedb...

Code Repositories


Code accompanying the paper "Interactively Learning Preference Constraints in Linear Bandits" (ICML 2022).

view repo

Please sign up or login with your details

Forgot password? Click here to reset