Can Public Large Language Models Help Private Cross-device Federated Learning?

by   Boxin Wang, et al.
University of Illinois at Urbana-Champaign

We study (differentially) private federated learning (FL) of language models. The language models in cross-device FL are relatively small, which can be trained with meaningful formal user-level differential privacy (DP) guarantees when massive parallelism in training is enabled by the participation of a moderate size of users. Recently, public data has been used to improve privacy-utility trade-offs for both large and small language models. In this work, we provide a systematic study of using large-scale public data and LLMs to help differentially private training of on-device FL models, and further improve the privacy-utility tradeoff by techniques of distillation. Moreover, we propose a novel distribution matching algorithm with theoretical grounding to sample public data close to private data distribution, which significantly improves the sample efficiency of (pre-)training on public data. The proposed method is efficient and effective for training private model by taking advantage of public data, especially for customized on-device architectures that do not have ready-to-use pre-trained models.


page 1

page 2

page 3

page 4


ULDP-FL: Federated Learning with Across Silo User-Level Differential Privacy

Differentially Private Federated Learning (DP-FL) has garnered attention...

Training Production Language Models without Memorizing User Data

This paper presents the first consumer-scale next-word prediction (NWP) ...

Differentially private federated deep learning for multi-site medical image segmentation

Collaborative machine learning techniques such as federated learning (FL...

OLIVE: Oblivious and Differentially Private Federated Learning on Trusted Execution Environment

Differentially private federated learning (DP-FL) has received increasin...

Differentially Private Distributed Learning for Language Modeling Tasks

One of the big challenges in machine learning applications is that train...

Impact of Sampling on Locally Differentially Private Data Collection

With the recent bloom of data, there is a huge surge in threats against ...

Federated Learning of Gboard Language Models with Differential Privacy

We train language models (LMs) with federated learning (FL) and differen...

Please sign up or login with your details

Forgot password? Click here to reset