Accelerated Federated Learning with Decoupled Adaptive Optimization

by   Jiayin Jin, et al.

The federated learning (FL) framework enables edge clients to collaboratively learn a shared inference model while keeping privacy of training data on clients. Recently, many heuristics efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings for improving convergence and accuracy. However, there is still a paucity of theoretical principles on where to and how to design and utilize adaptive optimization methods in federated settings. This work aims to develop novel adaptive optimization methods for FL from the perspective of dynamics of ordinary differential equations (ODEs). First, an analytic framework is established to build a connection between federated optimization methods and decompositions of ODEs of corresponding centralized optimizers. Second, based on this analytic framework, a momentum decoupling adaptive optimization method, FedDA, is developed to fully utilize the global momentum on each local iteration and accelerate the training convergence. Last but not least, full batch gradients are utilized to mimic centralized optimization in the end of the training process to ensure the convergence and overcome the possible inconsistency caused by adaptive optimization methods.


page 1

page 2

page 3

page 4


Local Adaptivity in Federated Learning: Convergence and Consistency

The federated learning (FL) framework trains a machine learning model us...

Adaptive Federated Optimization

Federated learning is a distributed machine learning paradigm in which a...

On Second-order Optimization Methods for Federated Learning

We consider federated learning (FL), where the training data is distribu...

Private Adaptive Optimization with Side Information

Adaptive optimization methods have become the default solvers for many m...

TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels

State-of-the-art federated learning methods can perform far worse than t...

Multi-task Federated Learning for Heterogeneous Pancreas Segmentation

Federated learning (FL) for medical image segmentation becomes more chal...

Please sign up or login with your details

Forgot password? Click here to reset