Distributed Adaptive Sampling for Kernel Matrix Approximation

03/27/2018
by   Daniele Calandriello, et al.
0

Most kernel-based methods, such as kernel or Gaussian process regression, kernel PCA, ICA, or k-means clustering, do not scale to large datasets, because constructing and storing the kernel matrix K_n requires at least O(n^2) time and space for n samples. Recent works show that sampling points with replacement according to their ridge leverage scores (RLS) generates small dictionaries of relevant points with strong spectral approximation guarantees for K_n. The drawback of RLS-based methods is that computing exact RLS requires constructing and storing the whole kernel matrix. In this paper, we introduce SQUEAK, a new algorithm for kernel approximation based on RLS sampling that sequentially processes the dataset, storing a dictionary which creates accurate kernel matrix approximations with a number of points that only depends on the effective dimension d_eff(γ) of the dataset. Moreover since all the RLS estimations are efficiently performed using only the small dictionary, SQUEAK is the first RLS sampling algorithm that never constructs the whole matrix K_n, runs in linear time O(nd_eff(γ)^3) w.r.t. n, and requires only a single pass over the dataset. We also propose a parallel and distributed version of SQUEAK that linearly scales across multiple machines, achieving similar accuracy in as little as O((n)d_eff(γ)^3) time.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/07/2017

Multiresolution Kernel Approximation for Gaussian Process Regression

Gaussian process regression generally does not scale to beyond a few tho...
research
02/23/2018

Exact Sampling of Determinantal Point Processes without Eigendecomposition

Determinantal point processes (DPPs) enable the modelling of repulsion: ...
research
05/24/2016

Recursive Sampling for the Nyström Method

We give the first algorithm for kernel Nyström approximation that runs i...
research
07/08/2020

Near Input Sparsity Time Kernel Embeddings via Adaptive Sampling

To accelerate kernel methods, we propose a near input sparsity time algo...
research
11/02/2014

Fast Randomized Kernel Methods With Statistical Guarantees

One approach to improving the running time of kernel-based machine learn...
research
01/15/2013

Why Size Matters: Feature Coding as Nystrom Sampling

Recently, the computer vision and machine learning community has been in...
research
10/06/2021

Coresets for Kernel Clustering

We devise the first coreset for kernel k-Means, and use it to obtain new...

Please sign up or login with your details

Forgot password? Click here to reset