Generalized Leverage Score Sampling for Neural Networks

by   Jason D. Lee, et al.

Leverage score sampling is a powerful technique that originates from theoretical computer science, which can be used to speed up a large number of fundamental questions, e.g. linear regression, linear programming, semi-definite programming, cutting plane method, graph sparsification, maximum matching and max-flow. Recently, it has been shown that leverage score sampling helps to accelerate kernel methods [Avron, Kapralov, Musco, Musco, Velingker and Zandieh 17]. In this work, we generalize the results in [Avron, Kapralov, Musco, Musco, Velingker and Zandieh 17] to a broader class of kernels. We further bring the leverage score sampling into the field of deep learning theory. ∙ We show the connection between the initialization for neural network training and approximating the neural tangent kernel with random features. ∙ We prove the equivalence between regularized neural network and neural tangent kernel ridge regression under the initialization of both classical random Gaussian and leverage score sampling.


page 1

page 2

page 3

page 4


On Fast Leverage Score Sampling and Optimal Learning

Leverage score sampling provides an appealing way to perform approximate...

Relating Leverage Scores and Density using Regularized Christoffel Functions

Statistical leverage scores emerged as a fundamental tool for matrix ske...

Scaling Neural Tangent Kernels via Sketching and Random Features

The Neural Tangent Kernel (NTK) characterizes the behavior of infinitely...

Tail bounds for volume sampled linear regression

The n × d design matrix in a linear regression problem is given, but the...

Gradient Coding through Iterative Block Leverage Score Sampling

We generalize the leverage score sampling sketch for ℓ_2-subspace embedd...

Random Features for the Neural Tangent Kernel

The Neural Tangent Kernel (NTK) has discovered connections between deep ...

Linearized two-layers neural networks in high dimension

We consider the problem of learning an unknown function f_ on the d-dime...

Please sign up or login with your details

Forgot password? Click here to reset