Mitigating Disparity while Maximizing Reward: Tight Anytime Guarantee for Improving Bandits

by   Vishakha Patil, et al.

We study the Improving Multi-Armed Bandit (IMAB) problem, where the reward obtained from an arm increases with the number of pulls it receives. This model provides an elegant abstraction for many real-world problems in domains such as education and employment, where decisions about the distribution of opportunities can affect the future capabilities of communities and the disparity between them. A decision-maker in such settings must consider the impact of her decisions on future rewards in addition to the standard objective of maximizing her cumulative reward at any time. In many of these applications, the time horizon is unknown to the decision-maker beforehand, which motivates the study of the IMAB problem in the technically more challenging horizon-unaware setting. We study the tension that arises between two seemingly conflicting objectives in the horizon-unaware setting: a) maximizing the cumulative reward at any time based on current rewards of the arms, and b) ensuring that arms with better long-term rewards get sufficient opportunities even if they initially have low rewards. We show that, surprisingly, the two objectives are aligned with each other in this setting. Our main contribution is an anytime algorithm for the IMAB problem that achieves the best possible cumulative reward while ensuring that the arms reach their true potential given sufficient time. Our algorithm mitigates the initial disparity due to lack of opportunity and continues pulling an arm till it stops improving. We prove the optimality of our algorithm by showing that a) any algorithm for the IMAB problem, no matter how utilitarian, must suffer Ω(T) policy regret and Ω(k) competitive ratio with respect to the optimal offline policy, and b) the competitive ratio of our algorithm is O(k).


page 1

page 2

page 3

page 4


Addressing the Long-term Impact of ML Decisions via Policy Regret

Machine Learning (ML) increasingly informs the allocation of opportuniti...

Blocking Bandits

We consider a novel stochastic multi-armed bandit setting, where playing...

Reinforcement Learning for Finite-Horizon Restless Multi-Armed Multi-Action Bandits

We study a finite-horizon restless multi-armed bandit problem with multi...

Dynamic Planning and Learning under Recovering Rewards

Motivated by emerging applications such as live-streaming e-commerce, pr...

Rotting Bandits

The Multi-Armed Bandits (MAB) framework highlights the tension between a...

Grooming a Single Bandit Arm

The stochastic multi-armed bandit problem captures the fundamental explo...

Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption

We study the infinite-horizon restless bandit problem with the average r...

Please sign up or login with your details

Forgot password? Click here to reset