DeepAI AI Chat
Log In Sign Up

On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology

by   Francesco Di Giovanni, et al.
University of Cambridge
University of Oxford
Sapienza University of Rome

Message Passing Neural Networks (MPNNs) are instances of Graph Neural Networks that leverage the graph to send messages over the edges. This inductive bias leads to a phenomenon known as over-squashing, where a node feature is insensitive to information contained at distant nodes. Despite recent methods introduced to mitigate this issue, an understanding of the causes for over-squashing and of possible solutions are lacking. In this theoretical work, we prove that: (i) Neural network width can mitigate over-squashing, but at the cost of making the whole network more sensitive; (ii) Conversely, depth cannot help mitigate over-squashing: increasing the number of layers leads to over-squashing being dominated by vanishing gradients; (iii) The graph topology plays the greatest role, since over-squashing occurs between nodes at high commute (access) time. Our analysis provides a unified framework to study different recent methods introduced to cope with over-squashing and serves as a justification for a class of methods that fall under `graph rewiring'.


page 1

page 2

page 3

page 4


Is Rewiring Actually Helpful in Graph Neural Networks?

Graph neural networks compute node representations by performing multipl...

How hard is graph isomorphism for graph neural networks?

A hallmark of graph neural networks is their ability to distinguish the ...

Unifying gradient regularization for Heterogeneous Graph Neural Networks

Heterogeneous Graph Neural Networks (HGNNs) are a class of powerful deep...

What graph neural networks cannot learn: depth vs width

This paper studies the capacity limits of graph neural networks (GNN). R...

Space4HGNN: A Novel, Modularized and Reproducible Platform to Evaluate Heterogeneous Graph Neural Network

Heterogeneous Graph Neural Network (HGNN) has been successfully employed...

Gradient scarcity with Bilevel Optimization for Graph Learning

A common issue in graph learning under the semi-supervised setting is re...

Topological Augmentation for Class-Imbalanced Node Classification

Class imbalance is prevalent in real-world node classification tasks and...