Federated learning using a mixture of experts

by   Edvin Listo Zec, et al.

Federated learning has received attention for its efficiency and privacy benefits, in settings where data is distributed among devices. Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current incarnations show limited privacy properties and have shortcomings when applied to common real-world scenarios. One such scenario is heterogeneous data among devices, where data may come from different generating distributions. In this paper, we propose a federated learning framework using a mixture of experts to balance the specialist nature of a locally trained model with the generalist knowledge of a global model in a federated learning setting. Our results show that the mixture of experts model is better suited as a personalized model for devices when data is heterogeneous, outperforming both global and local models. Furthermore, our framework gives strict privacy guarantees, which allows clients to select parts of their data that may be excluded from the federation. The evaluation shows that the proposed solution is robust to the setting where some users require a strict privacy setting and do not disclose their models to a central server at all, opting out from the federation partially or entirely. The proposed framework is general enough to include any kind of machine learning models, and can even use combinations of different kinds.


page 1

page 2

page 3

page 4


Adaptive Expert Models for Personalization in Federated Learning

Federated Learning (FL) is a promising framework for distributed learnin...

Comparative assessment of federated and centralized machine learning

Federated Learning (FL) is a privacy preserving machine learning scheme,...

Federated Mixture of Experts

Federated learning (FL) has emerged as the predominant approach for coll...

Lithography Hotspot Detection via Heterogeneous Federated Learning with Local Adaptation

As technology scaling is approaching the physical limit, lithography hot...

Collaborative Domain Blocking: Using federated NLP To Detect Malicious Domains

Current content filtering and blocking methods are susceptible to variou...

FedGen: Generalizable Federated Learning

Existing federated learning models that follow the standard risk minimiz...

Echo of Neighbors: Privacy Amplification for Personalized Private Federated Learning with Shuffle Model

Federated Learning, as a popular paradigm for collaborative training, is...

Please sign up or login with your details

Forgot password? Click here to reset