Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity

by   Bo Li, et al.

In federated learning, data heterogeneity is a critical challenge. A straightforward solution is to shuffle the clients' data to homogenize the distribution. However, this may violate data access rights, and how and when shuffling can accelerate the convergence of a federated optimization algorithm is not theoretically well understood. In this paper, we establish a precise and quantifiable correspondence between data heterogeneity and parameters in the convergence rate when a fraction of data is shuffled across clients. We prove that shuffling can quadratically reduce the gradient dissimilarity with respect to the shuffling percentage, accelerating convergence. Inspired by the theory, we propose a practical approach that addresses the data access rights issue by shuffling locally generated synthetic data. The experimental results show that shuffling synthetic data improves the performance of multiple existing federated learning algorithms by a large margin.


page 8

page 14

page 16

page 17


Federated Learning in Temporal Heterogeneity

In this work, we explored federated learning in temporal heterogeneity a...

Federated Learning via Synthetic Data

Federated learning allows for the training of a model using data on mult...

Fairness and Accuracy in Federated Learning

In the federated learning setting, multiple clients jointly train a mode...

Achieving Linear Convergence in Federated Learning under Objective and Systems Heterogeneity

We consider a standard federated learning architecture where a group of ...

Efficient Algorithms for Federated Saddle Point Optimization

We consider strongly convex-concave minimax problems in the federated se...

Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs

Decentralized and federated learning algorithms face data heterogeneity ...

Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning

Data heterogeneity has been identified as one of the key features in fed...

Please sign up or login with your details

Forgot password? Click here to reset