An Empirical and Comparative Analysis of Data Valuation with Scalable Algorithms

by   Ruoxi Jia, et al.
Shanghai Jiao Tong University
ETH Zurich
berkeley college

This paper focuses on valuating training data for supervised learning tasks and studies the Shapley value, a data value notion originated in cooperative game theory. The Shapley value defines a unique value distribution scheme that satisfies a set of appealing properties desired by a data value notion. However, the Shapley value requires exponential complexity to calculate exactly. Existing approximation algorithms, although achieving great improvement over the exact algorithm, relies on retraining models for multiple times, thus remaining limited when applied to larger-scale learning tasks and real-world datasets. In this work, we develop a simple and efficient heuristic for data valuation based on the Shapley value with complexity independent with the model size. The key idea is to approximate the model via a K-nearest neighbor (KNN) classifier, which has a locality structure that can lead to efficient Shapley value calculation. We evaluate the utility of the values produced by the KNN proxies in various settings, including label noise correction, watermark detection, data summarization, active data acquisition, and domain adaption. Extensive experiments demonstrate that our algorithm achieves at least comparable utility to the values produced by existing algorithms while significant efficiency improvement. Moreover, we theoretically analyze the Shapley value and justify its advantage over the leave-one-out error as a data value measure.


page 1

page 2

page 3

page 4


Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms

Given a data set D containing millions of data points and a data consume...

Towards Efficient Data Valuation Based on the Shapley Value

"How much is my data worth?" is an increasingly common question posed by...

A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"

Data valuation is a growing research field that studies the influence of...

Approximating the Shapley Value without Marginal Contributions

The Shapley value is arguably the most popular approach for assigning a ...

Data Banzhaf: A Data Valuation Framework with Maximal Robustness to Learning Stochasticity

This paper studies the robustness of data valuation to noisy model perfo...

Optimizing Data Shapley Interaction Calculation from O(2^n) to O(t n^2) for KNN models

With the rapid growth of data availability and usage, quantifying the ad...

Private Data Valuation and Fair Payment in Data Marketplaces

Data valuation is an essential task in a data marketplace. It aims at fa...

Please sign up or login with your details

Forgot password? Click here to reset