PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

by   Chulin Xie, et al.

Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high communication and computation costs or (2) overfit to local data, which can be limited in scope, and are vulnerable to evolved test samples with natural shifts. In this paper, we propose PerAda, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts. PerAda reduces the costs by leveraging the power of pretrained models and only updates and communicates a small number of additional parameters from adapters. PerAda has good generalization since it regularizes each client's personalized adapter with a global adapter, while the global adapter uses knowledge distillation to aggregate generalized information from all clients. Theoretically, we provide generalization bounds to explain why PerAda improves generalization, and we prove its convergence to stationary points under non-convex settings. Empirically, PerAda demonstrates competitive personalized performance (+4.85 out-of-distribution generalization (+5.23 across natural and medical domains compared with baselines, while only updating 12.6


page 1

page 2

page 3

page 4


Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning

Recently, data heterogeneity among the training datasets on the local cl...

Federated Learning under Covariate Shifts with Generalization Guarantees

This paper addresses intra-client and inter-client covariate shifts in f...

FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning

Federated learning (FL) has emerged as a new paradigm for privacy-preser...

Test-Time Robust Personalization for Federated Learning

Federated Learning (FL) is a machine learning paradigm where many client...

Federated Learning of Shareable Bases for Personalization-Friendly Image Classification

Personalized federated learning (PFL) aims to harness the collective wis...

FedHealth 2: Weighted Federated Transfer Learning via Batch Normalization for Personalized Healthcare

The success of machine learning applications often needs a large quantit...

Please sign up or login with your details

Forgot password? Click here to reset