Deconstructing Distributions: A Pointwise Framework of Learning

by   Gal Kaplun, et al.

In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the performance of a collection of models when evaluated on a single input point. Specifically, we study a point's profile: the relationship between models' average performance on the test distribution and their pointwise performance on this individual point. We find that profiles can yield new insights into the structure of both models and data – in and out-of-distribution. For example, we empirically show that real data distributions consist of points with qualitatively different profiles. On one hand, there are "compatible" points with strong correlation between the pointwise and average performance. On the other hand, there are points with weak and even negative correlation: cases where improving overall model accuracy actually hurts performance on these inputs. We prove that these experimental observations are inconsistent with the predictions of several simplified models of learning proposed in prior work. As an application, we use profiles to construct a dataset we call CIFAR-10-NEG: a subset of CINIC-10 such that for standard models, accuracy on CIFAR-10-NEG is negatively correlated with accuracy on CIFAR-10 test. This illustrates, for the first time, an OOD dataset that completely inverts "accuracy-on-the-line" (Miller, Taori, Raghunathan, Sagawa, Koh, Shankar, Liang, Carmon, and Schmidt 2021)


page 7

page 8

page 9

page 17

page 18

page 21


Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Machine learning is currently dominated by largely experimental work foc...

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

For machine learning systems to be reliable, we must understand their pe...

On the Unreported-Profile-is-Negative Assumption for Predictive Cheminformatics

In cheminformatics, compound-target binding profiles has been a main sou...

Enhancing the Diversity of Predictions Combination by Negative Correlation Learning

Predictions combination, as a combination model approach with adjustment...

Comparison of pharmacist evaluation of medication orders with predictions of a machine learning model

The objective of this work was to assess the clinical performance of an ...

Similarity learning for wells based on logging data

One of the first steps during the investigation of geological objects is...

Improving Baselines in the Wild

We share our experience with the recently released WILDS benchmark, a co...

Please sign up or login with your details

Forgot password? Click here to reset