Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks

by   Kenta Oono, et al.

It is known that the current graph neural networks (GNNs) are difficult to make themselves deep due to the problem known as over-smoothing. Multi-scale GNNs are a promising approach for mitigating the over-smoothing problem. However, there is little explanation of why it works empirically from the viewpoint of learning theory. In this study, we derive the optimization and generalization guarantees of transductive learning algorithms that include multi-scale GNNs. Using the boosting theory, we prove the convergence of the training error under weak learning-type conditions. By combining it with generalization gap bounds in terms of transductive Rademacher complexity, we show that a test error bound of a specific type of multi-scale GNNs that decreases corresponding to the depth under the conditions. Our results offer theoretical explanations for the effectiveness of the multi-scale structure against the over-smoothing problem. We apply boosting algorithms to the training of multi-scale GNNs for real-world node prediction tasks. We confirm that its performance is comparable to existing GNNs, and the practical behaviors are consistent with theoretical observations. Code is available at https://github.com/delta2323/GB-GNN


page 1

page 2

page 3

page 4


A Survey on Oversmoothing in Graph Neural Networks

Node features of graph neural networks (GNNs) tend to become more simila...

Elastic Graph Neural Networks

While many existing graph neural networks (GNNs) have been proven to per...

Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth

Graph Neural Networks (GNNs) have been studied through the lens of expre...

Transfer learning for atomistic simulations using GNNs and kernel mean embeddings

Interatomic potentials learned using machine learning methods have been ...

Analysis of Graph Neural Networks with Theory of Markov Chains

In this paper, we provide a theoretical tool for the interpretation and ...

How Can Graph Neural Networks Help Document Retrieval: A Case Study on CORD19 with Concept Map Generation

Graph neural networks (GNNs), as a group of powerful tools for represent...

Graph Neural Networks for Wireless Communications: From Theory to Practice

Deep learning-based approaches have been developed to solve challenging ...

Please sign up or login with your details

Forgot password? Click here to reset