Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transport

by   Jiying Zhang, et al.

Recently, the pretrain-finetuning paradigm has attracted tons of attention in graph learning community due to its power of alleviating the lack of labels problem in many real-world applications. Current studies use existing techniques, such as weight constraint, representation constraint, which are derived from images or text data, to transfer the invariant knowledge from the pre-train stage to fine-tuning stage. However, these methods failed to preserve invariances from graph structure and Graph Neural Network (GNN) style models. In this paper, we present a novel optimal transport-based fine-tuning framework called GTOT-Tuning, namely, Graph Topology induced Optimal Transport fine-Tuning, for GNN style backbones. GTOT-Tuning is required to utilize the property of graph data to enhance the preservation of representation produced by fine-tuned networks. Toward this goal, we formulate graph local knowledge transfer as an Optimal Transport (OT) problem with a structural prior and construct the GTOT regularizer to constrain the fine-tuned model behaviors. By using the adjacency relationship amongst nodes, the GTOT regularizer achieves node-level optimal transport procedures and reduces redundant transport procedures, resulting in efficient knowledge transfer from the pre-trained models. We evaluate GTOT-Tuning on eight downstream tasks with various GNN backbones and demonstrate that it achieves state-of-the-art fine-tuning performance for GNNs.


page 4

page 16

page 23

page 24


Prompt Tuning for Graph Neural Networks

In recent years, prompt tuning has set off a research boom in the adapta...

Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks

Recently, graph neural networks (GNNs) have shown its unprecedented succ...

AdapterGNN: Efficient Delta Tuning Improves Generalization Ability in Graph Neural Networks

Fine-tuning pre-trained models has recently yielded remarkable performan...

Optimal Transport Graph Neural Networks

Current graph neural network (GNN) architectures naively average or sum ...

Fine-tuning can cripple your foundation model; preserving features may be the solution

Pre-trained foundation models, owing primarily to their enormous capacit...

Visual Feature Encoding for GNNs on Road Networks

In this work, we present a novel approach to learning an encoding of vis...

Representation Transfer by Optimal Transport

Deep learning currently provides the best representations of complex obj...

Please sign up or login with your details

Forgot password? Click here to reset