GraphWorld: Fake Graphs Bring Real Insights for GNNs

by   John Palowitch, et al.

Despite advances in the field of Graph Neural Networks (GNNs), only a small number ( 5) of datasets are currently used to evaluate new models. This continued reliance on a handful of datasets provides minimal insight into the performance differences between models, and is especially challenging for industrial practitioners who are likely to have datasets which look very different from those used as academic benchmarks. In the course of our work on GNN infrastructure and open-source software at Google, we have sought to develop improved benchmarks that are robust, tunable, scalable,and generalizable. In this work we introduce GraphWorld, a novel methodology and system for benchmarking GNN models on an arbitrarily-large population of synthetic graphs for any conceivable GNN task. GraphWorld allows a user to efficiently generate a world with millions of statistically diverse datasets. It is accessible, scalable, and easy to use. GraphWorld can be run on a single machine without specialized hardware, or it can be easily scaled up to run on arbitrary clusters or cloud frameworks. Using GraphWorld, a user has fine-grained control over graph generator parameters, and can benchmark arbitrary GNN models with built-in hyperparameter tuning. We present insights from GraphWorld experiments regarding the performance characteristics of tens of thousands of GNN models over millions of benchmark datasets. We further show that GraphWorld efficiently explores regions of benchmark dataset space uncovered by standard benchmarks, revealing comparisons between models that have not been historically obtainable. Using GraphWorld, we also are able to study in-detail the relationship between graph properties and task performance metrics, which is nearly impossible with the classic collection of real-world benchmarks.


page 1

page 11


Evaluating Explainability for Graph Neural Networks

As post hoc explanations are increasingly used to understand the behavio...

Towards a Taxonomy of Graph Learning Datasets

Graph neural networks (GNNs) have attracted much attention due to their ...

Taxonomy of Benchmarks in Graph Representation Learning

Graph Neural Networks (GNNs) extend the success of neural networks to gr...

Search for the UGLE Truth: An Investigation into Unsupervised GNN Learning Environments

Graph Neural Networks (GNNs) are a pertinent tool for any machine learni...

Beyond Real-world Benchmark Datasets: An Empirical Study of Node Classification with GNNs

Graph Neural Networks (GNNs) have achieved great success on a node class...

gSuite: A Flexible and Framework Independent Benchmark Suite for Graph Neural Network Inference on GPUs

As the interest to Graph Neural Networks (GNNs) is growing, the importan...

Scalable Verification of GNN-based Job Schedulers

Recently, Graph Neural Networks (GNNs) have been applied for scheduling ...

Please sign up or login with your details

Forgot password? Click here to reset