FedBone: Towards Large-Scale Federated Multi-Task Learning

by   Yiqiang Chen, et al.
Institute of Computing Technology, Chinese Academy of Sciences

Heterogeneous federated multi-task learning (HFMTL) is a federated learning technique that combines heterogeneous tasks of different clients to achieve more accurate, comprehensive predictions. In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features. However, large-scale models cannot be directly applied to existing federated multi-task learning methods. Existing HFML methods also disregard the impact of gradient conflicts on multi-task optimization during the federated aggregation process. In this work, we propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization from the perspective of server-client split learning and gradient projection. We split the entire model into two components: a large-scale general model (referred to as the general model) on the cloud server and multiple task-specific models (referred to as the client model) on edge clients, solving the problem of insufficient computing power on edge clients. The conflicting gradient projection technique is used to enhance the generalization of the large-scale general model between different tasks. The proposed framework is evaluated on two benchmark datasets and a real ophthalmic dataset. Comprehensive results demonstrate that FedBone efficiently adapts to heterogeneous local tasks of each client and outperforms existing federated learning algorithms in most dense prediction and classification tasks with off-the-shelf computational resources on the client side.


page 3

page 7


Variational Federated Multi-Task Learning

In classical federated learning a central server coordinates the trainin...

FewFedWeight: Few-shot Federated Learning Framework across Multiple NLP Tasks

Massively multi-task learning with large language models has recently ma...

Performance Optimization for Federated Person Re-identification via Benchmark Analysis

Federated learning is a privacy-preserving machine learning technique th...

Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training

Federated learning, which shares the weights of the neural network acros...

Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning

We introduce a library, Dataset Grouper, to create large-scale group-str...

Networked Federated Multi-Task Learning

Many important application domains generate distributed collections of h...

SplitGP: Achieving Both Generalization and Personalization in Federated Learning

A fundamental challenge to providing edge-AI services is the need for a ...

Please sign up or login with your details

Forgot password? Click here to reset