DeepAI AI Chat
Log In Sign Up

Unifying Approaches in Data Subset Selection via Fisher Information and Information-Theoretic Quantities

by   Andreas Kirsch, et al.

The mutual information between predictions and model parameters – also referred to as expected information gain or BALD in machine learning – measures informativeness. It is a popular acquisition function in Bayesian active learning and Bayesian optimal experiment design. In data subset selection, i.e. active learning and active sampling, several recent works use Fisher information, Hessians, similarity matrices based on the gradients, or simply the gradient lengths to compute the acquisition scores that guide sample selection. Are these different approaches connected, and if so how? In this paper, we revisit the Fisher information and use it to show how several otherwise disparate methods are connected as approximations of information-theoretic quantities.


page 1

page 2

page 3

page 4


A Practical Unified Notation for Information-Theoretic Quantities in ML

Information theory is of importance to machine learning, but the notatio...

Prediction-Oriented Bayesian Active Learning

Information-theoretic approaches to active learning have traditionally f...

An Information-Theoretic Framework for Unifying Active Learning Problems

This paper presents an information-theoretic framework for unifying acti...

Explaining Predictive Uncertainty with Information Theoretic Shapley Values

Researchers in explainable artificial intelligence have developed numero...

Active information requirements for fixation on the Wright-Fisher model of population genetics

In the context of population genetics, active information can be extende...

Prioritized training on points that are learnable, worth learning, and not yet learned

We introduce Goldilocks Selection, a technique for faster model training...

Information-Theoretic Active Learning for Content-Based Image Retrieval

We propose Information-Theoretic Active Learning (ITAL), a novel batch-m...