Fast Graph Neural Tangent Kernel via Kronecker Sketching

by   Shunhua Jiang, et al.

Many deep learning tasks have to deal with graphs (e.g., protein structures, social networks, source code abstract syntax trees). Due to the importance of these tasks, people turned to Graph Neural Networks (GNNs) as the de facto method for learning on graphs. GNNs have become widely applied due to their convincing performance. Unfortunately, one major barrier to using GNNs is that GNNs require substantial time and resources to train. Recently, a new method for learning on graph data is Graph Neural Tangent Kernel (GNTK) [Du, Hou, Salakhutdinov, Poczos, Wang and Xu 19]. GNTK is an application of Neural Tangent Kernel (NTK) [Jacot, Gabriel and Hongler 18] (a kernel method) on graph data, and solving NTK regression is equivalent to using gradient descent to train an infinite-wide neural network. The key benefit of using GNTK is that, similar to any kernel method, GNTK's parameters can be solved directly in a single step. This can avoid time-consuming gradient descent. Meanwhile, sketching has become increasingly used in speeding up various optimization problems, including solving kernel regression. Given a kernel matrix of n graphs, using sketching in solving kernel regression can reduce the running time to o(n^3). But unfortunately such methods usually require extensive knowledge about the kernel matrix beforehand, while in the case of GNTK we find that the construction of the kernel matrix is already O(n^2N^4), assuming each graph has N nodes. The kernel matrix construction time can be a major performance bottleneck when the size of graphs N increases. A natural question to ask is thus whether we can speed up the kernel matrix construction to improve GNTK regression's end-to-end running time. This paper provides the first algorithm to construct the kernel matrix in o(n^2N^3) running time.


page 1

page 2

page 3

page 4


Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?

A rising trend in theoretical deep learning is to understand why deep le...

Graph Neural Tangent Kernel: Convergence on Large Graphs

Graph neural networks (GNNs) achieve remarkable performance in graph mac...

Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels

While graph kernels (GKs) are easy to train and enjoy provable theoretic...

Adaptive fractional order graph neural network

This paper proposes adaptive fractional order graph neural network (AFGN...

Graph Construction from Data using Non Negative Kernel regression (NNK Graphs)

Data driven graph constructions are often used in various applications, ...

PIGEON: Optimizing CUDA Code Generator for End-to-End Training and Inference of Relational Graph Neural Networks

Relational graph neural networks (RGNNs) are graph neural networks (GNNs...

Graph Fuzzy System: Concepts, Models and Algorithms

Fuzzy systems (FSs) have enjoyed wide applications in various fields, in...

Please sign up or login with your details

Forgot password? Click here to reset