DeepAI AI Chat
Log In Sign Up

SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training

04/15/2023
by   Riyasat Ohib, et al.
1

Federated learning (FL) enables the training of a model leveraging decentralized data in client sites while preserving privacy by not collecting data. However, one of the significant challenges of FL is limited computation and low communication bandwidth in resource limited edge client nodes. To address this, several solutions have been proposed in recent times including transmitting sparse models and learning dynamic masks iteratively, among others. However, many of these methods rely on transmitting the model weights throughout the entire training process as they are based on ad-hoc or random pruning criteria. In this work, we propose Salient Grads, which simplifies the process of sparse training by choosing a data aware subnetwork before training, based on the model-parameter's saliency scores, which is calculated from the local client data. Moreover only highly sparse gradients are transmitted between the server and client models during the training process unlike most methods that rely on sharing the entire dense model in each round. We also demonstrate the efficacy of our method in a real world federated learning application and report improvement in wall-clock communication time.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/03/2022

GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learning

Federated learning (FL) is an effective technique to directly involve ed...
03/30/2023

DPP-based Client Selection for Federated Learning with Non-IID Data

This paper proposes a client selection (CS) method to tackle the communi...
12/18/2021

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Federated learning (FL) enables distribution of machine learning workloa...
05/24/2022

Wireless Ad Hoc Federated Learning: A Fully Distributed Cooperative Machine Learning

Federated learning has allowed training of a global model by aggregating...
06/09/2022

HideNseek: Federated Lottery Ticket via Server-side Pruning and Sign Supermask

Federated learning alleviates the privacy risk in distributed learning b...
03/25/2022

Sparse Federated Learning with Hierarchical Personalization Models

Federated learning (FL) is widely used in the Internet of Things (IoT), ...