A Theory of Usable Information Under Computational Constraints

by   Yilun Xu, et al.
Peking University
Stanford University

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.


page 1

page 2

page 3

page 4


Generalized Mutual Information

Mutual information is one of the essential building blocks of informatio...

High Mutual Information in Representation Learning with Symmetric Variational Inference

We introduce the Mutual Information Machine (MIM), a novel formulation o...

Deep Sufficient Representation Learning via Mutual Information

We propose a mutual information-based sufficient representation learning...

Invariant Representations with Stochastically Quantized Neural Networks

Representation learning algorithms offer the opportunity to learn invari...

FSMI: Fast computation of Shannon Mutual Information for information-theoretic mapping

Exploration tasks are embedded in many robotics applications, such as se...

Representation Learning with Information Theory for COVID-19 Detection

Successful data representation is a fundamental factor in machine learni...

Information, learning and falsification

There are (at least) three approaches to quantifying information. The fi...

Please sign up or login with your details

Forgot password? Click here to reset