Adaptive Federated Optimization

by   Sashank Reddi, et al.

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Due to the heterogeneity of the client datasets, standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general nonconvex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.


page 1

page 2

page 3

page 4


Communication-Efficient Adaptive Federated Learning

Federated learning is a machine learning training paradigm that enables ...

FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data

Federated learning is an emerging distributed machine learning method, e...

Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

Federated learning allows loads of edge computing devices to collaborati...

Accelerated Federated Learning with Decoupled Adaptive Optimization

The federated learning (FL) framework enables edge clients to collaborat...

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning

An oft-cited challenge of federated learning is the presence of heteroge...

FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participation

Federated learning is a framework for distributed optimization that plac...

On the Convergence of Federated Optimization in Heterogeneous Networks

The burgeoning field of federated learning involves training machine lea...

Code Repositories



view repo

Please sign up or login with your details

Forgot password? Click here to reset