Prediction-Oriented Bayesian Active Learning

04/17/2023
by   Freddie Bickford Smith, et al.
0

Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.

READ FULL TEXT
research
08/01/2022

Unifying Approaches in Data Subset Selection via Fisher Information and Information-Theoretic Quantities

The mutual information between predictions and model parameters – also r...
research
05/27/2022

Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias

Bayesian adaptive experimental design is a form of active learning, whic...
research
12/24/2011

Bayesian Active Learning for Classification and Preference Learning

Information theoretic active learning has been widely studied for probab...
research
05/30/2021

BABA: Beta Approximation for Bayesian Active Learning

This paper introduces a new acquisition function under the Bayesian acti...
research
06/20/2012

On Discarding, Caching, and Recalling Samples in Active Learning

We address challenges of active learning under scarce informational reso...
research
10/04/2022

Active Learning for Regression with Aggregated Outputs

Due to the privacy protection or the difficulty of data collection, we c...
research
06/09/2023

Explaining Predictive Uncertainty with Information Theoretic Shapley Values

Researchers in explainable artificial intelligence have developed numero...

Please sign up or login with your details

Forgot password? Click here to reset