FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

by   Han Guo, et al.

Influence functions approximate the 'influences' of training data-points for test predictions and have a wide variety of applications. Despite the popularity, their computational cost does not scale well with model and training data size. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. We use k-Nearest Neighbors (kNN) to narrow the search space down to a subset of good candidate data points, identify the configurations that best balance the speed-quality trade-off in estimating the inverse Hessian-vector product, and introduce a fast parallel variant. Our proposed method achieves about 80x speedup while being highly correlated with the original influence values. With the availability of the fast influence functions, we demonstrate their usefulness in four applications. First, we examine whether influential data-points can 'explain' test time behavior using the framework of simulatability. Second, we visualize the influence interactions between training and test data-points. Third, we show that we can correct model errors by additional fine-tuning on certain influential data-points, improving the accuracy of a trained MNLI model by 2.6 small number of gradient updates. Finally, we experiment with a data-augmentation setup where we use influence functions to search for new data-points unseen during training to improve model performance. Overall, our fast influence functions can be efficiently applied to large models and datasets, and our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors. Code is available at


page 4

page 5

page 7

page 12

page 13

page 14

page 15

page 18


Scaling Up Influence Functions

We address efficient calculation of influence functions for tracking pre...

Class based Influence Functions for Error Detection

Influence functions (IFs) are a powerful tool for detecting anomalous ex...

Rewarding High-Quality Data via Influence Functions

We consider a crowdsourcing data acquisition scenario, such as federated...

Interpreting Deep Models through the Lens of Data

Identification of input data points relevant for the classifier (i.e. se...

Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value

Data valuation is a powerful framework for providing statistical insight...

Studying Large Language Model Generalization with Influence Functions

When trying to gain better visibility into a machine learning model in o...

HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks

The behaviors of deep neural networks (DNNs) are notoriously resistant t...

Please sign up or login with your details

Forgot password? Click here to reset