TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels

by   Yaodong Yu, et al.

State-of-the-art federated learning methods can perform far worse than their centralized counterparts when clients have dissimilar data distributions. For neural networks, even when centralized SGD easily finds a solution that is simultaneously performant for all clients, current federated optimization methods fail to converge to a comparable solution. We show that this performance disparity can largely be attributed to optimization challenges presented by nonconvexity. Specifically, we find that the early layers of the network do learn useful features, but the final layers fail to make use of them. That is, federated optimization applied to this non-convex problem distorts the learning of the final layers. Leveraging this observation, we propose a Train-Convexify-Train (TCT) procedure to sidestep this issue: first, learn features using off-the-shelf methods (e.g., FedAvg); then, optimize a convexified problem obtained from the network's empirical neural tangent kernel approximation. Our technique yields accuracy improvements of up to +36 FMNIST and +37


page 1

page 2

page 3

page 4


Federated Residual Learning

We study a new form of federated learning where the clients train person...

On Large-Cohort Training for Federated Learning

Federated learning methods typically learn a model by iteratively sampli...

Multi-objective Evolutionary Federated Learning

Federated learning is an emerging technique used to prevent the leakage ...

Accelerated Federated Learning with Decoupled Adaptive Optimization

The federated learning (FL) framework enables edge clients to collaborat...

Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients

Federated optimization, an emerging paradigm which finds wide real-world...

Learning Federated Representations and Recommendations with Limited Negatives

Deep retrieval models are widely used for learning entity representation...

Compare Where It Matters: Using Layer-Wise Regularization To Improve Federated Learning on Heterogeneous Data

Federated Learning is a widely adopted method to train neural networks o...

Please sign up or login with your details

Forgot password? Click here to reset