Bi-level Multi-objective Evolutionary Learning: A Case Study on Multi-task Graph Neural Topology Search

by   Chao Wang, et al.

The construction of machine learning models involves many bi-level multi-objective optimization problems (BL-MOPs), where upper level (UL) candidate solutions must be evaluated via training weights of a model in the lower level (LL). Due to the Pareto optimality of sub-problems and the complex dependency across UL solutions and LL weights, an UL solution is feasible if and only if the LL weight is Pareto optimal. It is computationally expensive to determine which LL Pareto weight in the LL Pareto weight set is the most appropriate for each UL solution. This paper proposes a bi-level multi-objective learning framework (BLMOL), coupling the above decision-making process with the optimization process of the UL-MOP by introducing LL preference r. Specifically, the UL variable and r are simultaneously searched to minimize multiple UL objectives by evolutionary multi-objective algorithms. The LL weight with respect to r is trained to minimize multiple LL objectives via gradient-based preference multi-objective algorithms. In addition, the preference surrogate model is constructed to replace the expensive evaluation process of the UL-MOP. We consider a novel case study on multi-task graph neural topology search. It aims to find a set of Pareto topologies and their Pareto weights, representing different trade-offs across tasks at UL and LL, respectively. The found graph neural network is employed to solve multiple tasks simultaneously, including graph classification, node classification, and link prediction. Experimental results demonstrate that BLMOL can outperform some state-of-the-art algorithms and generate well-representative UL solutions and LL weights.


Self-Evolutionary Optimization for Pareto Front Learning

Multi-task learning (MTL), which aims to improve performance by learning...

End-to-End Pareto Set Prediction with Graph Neural Networks for Multi-objective Facility Location

The facility location problems (FLPs) are a typical class of NP-hard com...

The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering

In presence of multiple objectives to be optimized in Search-Based Softw...

A Two-phase Framework with a Bézier Simplex-based Interpolation Method for Computationally Expensive Multi-objective Optimization

This paper proposes a two-phase framework with a Bézier simplex-based in...

Hierarchical Bi-level Multi-Objective Evolution of Single- and Multi-layer Echo State Network Autoencoders for Data Representations

Echo State Network (ESN) presents a distinguished kind of recurrent neur...

Mind the Gap: Measuring Generalization Performance Across Multiple Objectives

Modern machine learning models are often constructed taking into account...

Inferring Gene Regulatory Network Using An Evolutionary Multi-Objective Method

Inference of gene regulatory networks (GRNs) based on experimental data ...

Please sign up or login with your details

Forgot password? Click here to reset