DIVA: Dataset Derivative of a Learning Task

by   Yonatan Dukler, et al.

We present a method to compute the derivative of a learning task with respect to a dataset. A learning task is a function from a training set to the validation error, which can be represented by a trained deep neural network (DNN). The "dataset derivative" is a linear operator, computed around the trained model, that informs how perturbations of the weight of each training sample affect the validation error, usually computed on a separate validation dataset. Our method, DIVA (Differentiable Validation) hinges on a closed-form differentiable expression of the leave-one-out cross-validation error around a pre-trained DNN. Such expression constitutes the dataset derivative. DIVA could be used for dataset auto-curation, for example removing samples with faulty annotations, augmenting a dataset with additional relevant samples, or rebalancing. More generally, DIVA can be used to optimize the dataset, along with the parameters of the model, as part of the training process without the need for a separate validation dataset, unlike bi-level optimization methods customary in AutoML. To illustrate the flexibility of DIVA, we report experiments on sample auto-curation tasks such as outlier rejection, dataset extension, and automatic aggregation of multi-modal data.


A Novel DNN Training Framework via Data Sampling and Multi-Task Optimization

Conventional DNN training paradigms typically rely on one training set a...

Towards Deep Network Steganography: From Networks to Networks

With the widespread applications of the deep neural network (DNN), how t...

On Optimal Generalizability in Parametric Learning

We consider the parametric learning problem, where the objective of the ...

Task-Driven Data Verification via Gradient Descent

We introduce a novel algorithm for the detection of possible sample corr...

Comparing Sample-wise Learnability Across Deep Neural Network Models

Estimating the relative importance of each sample in a training set has ...

Managing dataset shift by adversarial validation for credit scoring

Dataset shift is common in credit scoring scenarios, and the inconsisten...

Please sign up or login with your details

Forgot password? Click here to reset