TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels

07/13/2022
by   Yaodong Yu, et al.
5

State-of-the-art federated learning methods can perform far worse than their centralized counterparts when clients have dissimilar data distributions. For neural networks, even when centralized SGD easily finds a solution that is simultaneously performant for all clients, current federated optimization methods fail to converge to a comparable solution. We show that this performance disparity can largely be attributed to optimization challenges presented by nonconvexity. Specifically, we find that the early layers of the network do learn useful features, but the final layers fail to make use of them. That is, federated optimization applied to this non-convex problem distorts the learning of the final layers. Leveraging this observation, we propose a Train-Convexify-Train (TCT) procedure to sidestep this issue: first, learn features using off-the-shelf methods (e.g., FedAvg); then, optimize a convexified problem obtained from the network's empirical neural tangent kernel approximation. Our technique yields accuracy improvements of up to +36 FMNIST and +37

READ FULL TEXT

page 1

page 2

page 3

page 4

03/28/2020

Federated Residual Learning

We study a new form of federated learning where the clients train person...
06/15/2021

On Large-Cohort Training for Federated Learning

Federated learning methods typically learn a model by iteratively sampli...
12/18/2018

Multi-objective Evolutionary Federated Learning

Federated learning is an emerging technique used to prevent the leakage ...
07/14/2022

Accelerated Federated Learning with Decoupled Adaptive Optimization

The federated learning (FL) framework enables edge clients to collaborat...
08/08/2023

Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients

Federated optimization, an emerging paradigm which finds wide real-world...
08/18/2021

Learning Federated Representations and Recommendations with Limited Negatives

Deep retrieval models are widely used for learning entity representation...
12/01/2021

Compare Where It Matters: Using Layer-Wise Regularization To Improve Federated Learning on Heterogeneous Data

Federated Learning is a widely adopted method to train neural networks o...

Please sign up or login with your details

Forgot password? Click here to reset