Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias

by   Sabina J. Sloman, et al.

Bayesian adaptive experimental design is a form of active learning, which chooses samples to maximize the information they give about uncertain parameters. Prior work has shown that other forms of active learning can suffer from active learning bias, where unrepresentative sampling leads to inconsistent parameter estimates. We show that active learning bias can also afflict Bayesian adaptive experimental design, depending on model misspecification. We develop an information-theoretic measure of misspecification, and show that worse misspecification implies more severe active learning bias. At the same time, model classes incorporating more "noise" - i.e., specifying higher inherent variance in observations - suffer less from active learning bias, because their predictive distributions are likely to overlap more with the true distribution. Finally, we show how these insights apply to a (simulated) preference learning experiment.


page 1

page 2

page 3

page 4


Mitigating Sampling Bias and Improving Robustness in Active Learning

This paper presents simple and efficient methods to mitigate sampling bi...

Bayesian Active Learning for Classification and Preference Learning

Information theoretic active learning has been widely studied for probab...

On Statistical Bias In Active Learning: How and When To Fix It

Active learning is a powerful tool when labelling data is expensive, but...

Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not

Farquhar et al. [2021] show that correcting for active learning bias wit...

Prediction-Oriented Bayesian Active Learning

Information-theoretic approaches to active learning have traditionally f...

Introducing Bayesian Analysis with m&m's^: an active-learning exercise for undergraduates

We present an active-learning strategy for undergraduates that applies B...

Automatic Playtesting for Game Parameter Tuning via Active Learning

Game designers use human playtesting to gather feedback about game desig...

Please sign up or login with your details

Forgot password? Click here to reset