Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

by   Chun-Yin Huang, et al.

Despite Federated Learning (FL)'s trend for learning machine learning models in a distributed manner, it is susceptible to performance drops when training on heterogeneous data. Recently, dataset distillation has been explored in order to improve the efficiency and scalability of FL by creating a smaller, synthetic dataset that retains the performance of a model trained on the local private datasets. We discover that using distilled local datasets can amplify the heterogeneity issue in FL. To address this, we propose a new method, called Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation (FEDLGD), which trains FL using a smaller synthetic dataset (referred as virtual data) created through a combination of local and global distillation. Specifically, to handle synchronization and class imbalance, we propose iterative distribution matching to allow clients to have the same amount of balanced local virtual data; to harmonize the domain shifts, we use federated gradient matching to distill global virtual data that are shared with clients without hindering data privacy to rectify heterogeneous local training via enforcing local-global feature similarity. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources. Our method outperforms state-of-the-art heterogeneous FL algorithms under the setting with a very limited amount of distilled virtual data.


page 4

page 12

page 13

page 14

page 15

page 20


Rethinking Client Drift in Federated Learning: A Logit Perspective

Federated Learning (FL) enables multiple clients to collaboratively lear...

CD^2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning

Federated learning (FL) is a distributed learning paradigm that enables ...

Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution

Federated medical relation extraction enables multiple clients to train ...

Federated Learning with Intermediate Representation Regularization

In contrast to centralized model training that involves data collection,...

FedRAD: Federated Robust Adaptive Distillation

The robustness of federated learning (FL) is vital for the distributed t...

FedSiam-DA: Dual-aggregated Federated Learning via Siamese Network under Non-IID Data

Federated learning is a distributed learning that allows each client to ...

FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence

As a distributed learning paradigm, Federated Learning (FL) faces the co...

Please sign up or login with your details

Forgot password? Click here to reset