Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning

by   Yun-Hin Chan, et al.

Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homogeneous FL methods in handling system heterogeneity, we propose a training scheme that can extend their capabilities to cope with this challenge. In this paper, we commence our study with a detailed exploration of homogeneous and heterogeneous FL settings and discover three key observations: (1) a positive correlation between client performance and layer similarities, (2) higher similarities in the shallow layers in contrast to the deep layers, and (3) the smoother gradients distributions indicate the higher layer similarities. Building upon these observations, we propose InCo Aggregation that leverags internal cross-layer gradients, a mixture of gradients from shallow and deep layers within a server model, to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, our methods can be tailored to accommodate model-homogeneous FL methods such as FedAvg, FedProx, FedNova, Scaffold, and MOON, to expand their capabilities to handle the system heterogeneity. Copious experimental results validate the effectiveness of InCo Aggregation, spotlighting internal cross-layer gradients as a promising avenue to enhance the performance in heterogenous FL.


page 19

page 20

page 21


Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning

The heterogeneity of hardware and data is a well-known and studied probl...

Addressing Heterogeneity in Federated Learning via Distributional Transformation

Federated learning (FL) allows multiple clients to collaboratively train...

Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization

In practical federated learning scenarios, the participating devices may...

Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning

In federated learning (FL), model performance typically suffers from cli...

Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training

Federated learning (FL) is a rapidly growing privacy-preserving collabor...

Improving Federated Aggregation with Deep Unfolding Networks

The performance of Federated learning (FL) is negatively affected by dev...

Please sign up or login with your details

Forgot password? Click here to reset