A Practical Unified Notation for Information-Theoretic Quantities in ML

06/22/2021
by   Andreas Kirsch, et al.
51

Information theory is of importance to machine learning, but the notation for information-theoretic quantities is sometimes opaque. The right notation can convey valuable intuitions and concisely express new ideas. We propose such a notation for machine learning users and expand it to include information-theoretic quantities between events (outcomes) and random variables. We apply this notation to a popular information-theoretic acquisition function in Bayesian active learning which selects the most informative (unlabelled) samples to be labelled by an expert. We demonstrate the value of our notation when extending the acquisition function to the core-set problem, which consists of selecting the most informative samples given the labels.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/01/2022

Unifying Approaches in Data Subset Selection via Fisher Information and Information-Theoretic Quantities

The mutual information between predictions and model parameters – also r...
research
12/16/2022

A unified information-theoretic model of EEG signatures of human language processing

We advance an information-theoretic model of human language processing i...
research
12/19/2020

An Information-Theoretic Framework for Unifying Active Learning Problems

This paper presents an information-theoretic framework for unifying acti...
research
12/19/2014

Information-Theoretic Methods for Identifying Relationships among Climate Variables

Information-theoretic quantities, such as entropy, are used to quantify ...
research
07/06/2021

Prioritized training on points that are learnable, worth learning, and not yet learned

We introduce Goldilocks Selection, a technique for faster model training...
research
12/14/2018

The Value of Interaction in Data Intelligence

In human computer interaction (HCI), it is common to evaluate the value ...
research
03/21/2022

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies

Studies investigating neural information processing often implicitly ask...

Please sign up or login with your details

Forgot password? Click here to reset