Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models

by   Sixing Yu, et al.

Foundation Models (FMs), such as BERT, GPT, ViT, and CLIP, have demonstrated remarkable success in a wide range of applications, driven by their ability to leverage vast amounts of data for pre-training. However, optimizing FMs often requires access to sensitive data, raising privacy concerns and limiting their applicability in certain domains. In this paper, we introduce the concept of Federated Foundation Models (FFMs), a novel approach that combines the benefits of FMs and Federated Learning (FL) to enable privacy-preserving and collaborative learning across multiple institutions. We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We further provide formal definitions of FFM tasks, including FFM pre-training, FFM fine-tuning, and federated prompt engineering, allowing for more personalized and context-aware models while maintaining data privacy. Moreover, we explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge unlocks the potential for optimizing FMs using newly generated private data at edges. We present experiments and evaluations comparing the performance of FFMs to traditional FMs on various downstream tasks, demonstrating the effectiveness of our approach in preserving privacy, reducing overfitting, and improving model generalizability. The proposed Federated Foundation Models offer a flexible and scalable framework for training large language models in a privacy-preserving manner, paving the way for future advancements in both FM pre-training and federated learning.


page 1

page 2

page 3

page 4


Federated Large Language Model: A Position Paper

Large scale language models (LLM) have received significant attention an...

Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Speech Emotion Recognition (SER) refers to the recognition of human emot...

FDAPT: Federated Domain-adaptive Pre-training for Language Models

Combining Domain-adaptive Pre-training (DAPT) with Federated Learning (F...

FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

LLMs have demonstrated great capabilities in various NLP tasks. Differen...

FedBot: Enhancing Privacy in Chatbots with Federated Learning

Chatbots are mainly data-driven and usually based on utterances that mig...

Towards Building the Federated GPT: Federated Instruction Tuning

While “instruction-tuned" generative large language models (LLMs) have d...

MS-DINO: Efficient Distributed Training of Vision Transformer Foundation Model in Medical Domain through Masked Sampling

In spite of the recent success of deep learning in the medical domain, t...

Please sign up or login with your details

Forgot password? Click here to reset