DeepAI AI Chat
Log In Sign Up

Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training

by   Yifan Shi, et al.

Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem–data heterogeneity in real FL systems. However, almost all existing works have to face large communication burdens and the risk of disruption if the central server fails. Only limited efforts have been used in a decentralized way but still suffers from inferior representation ability due to sharing the full model with its neighbors. Therefore, in this paper, we propose a personalized FL framework with a decentralized partial model training called DFedAlt. It personalizes the "right" components in the modern deep models by alternately updating the shared and personal parameters to train partially personalized models in a peer-to-peer manner. To further promote the shared parameters aggregation process, we propose DFedSalt integrating the local Sharpness Aware Minimization (SAM) optimizer to update the shared parameters. It adds proper perturbation in the direction of the gradient to overcome the shared model inconsistency across clients. Theoretically, we provide convergence analysis of both algorithms in the general non-convex setting for decentralized partial model training in PFL. Our experiments on several real-world data with various data partition settings demonstrate that (i) decentralized training is more suitable for partial personalization, which results in state-of-the-art (SOTA) accuracy compared with the SOTA PFL baselines; (ii) the shared parameters with proper perturbation make partial personalized FL more suitable for decentralized training, where DFedSalt achieves most competitive performance.


page 1

page 2

page 3

page 4


Improving the Model Consistency of Decentralized Federated Learning

To mitigate the privacy leakages and communication burdens of Federated ...

Federated Learning with Partial Model Personalization

We consider two federated learning algorithms for training partially per...

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Personalized federated learning is proposed to handle the data heterogen...

Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

We present a partially personalized formulation of Federated Learning (F...

A Local Updating Algorithm for Personalized PageRank via Chebyshev Polynomials

The personalized PageRank algorithm is one of the most versatile tools f...

Decentralized Bayesian Learning over Graphs

We propose a decentralized learning algorithm over a general social netw...

Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs

Decentralized and federated learning algorithms face data heterogeneity ...