FedScale: Benchmarking Model and System Performance of Federated Learning at Scale

by   Fan Lai, et al.

We present FedScale, a diverse set of challenging and realistic benchmark datasets to facilitate scalable, comprehensive, and reproducible federated learning (FL) research. FedScale datasets are large-scale, encompassing a diverse range of important FL tasks, such as image classification, object detection, word prediction, and speech recognition. For each dataset, we provide a unified evaluation protocol using realistic data splits and evaluation metrics. To meet the pressing need for reproducing realistic FL at scale, we have also built an efficient evaluation platform to simplify and standardize the process of FL experimental setup and model evaluation. Our evaluation platform provides flexible APIs to implement new FL algorithms and includes new execution backends with minimal developer efforts. Finally, we perform in-depth benchmark experiments on these datasets. Our experiments suggest fruitful opportunities in heterogeneity-aware co-optimizations of the system and statistical efficiency under realistic FL characteristics. FedScale is open-source with permissive licenses and actively maintained, and we welcome feedback and contributions from the community.


A Survey for Federated Learning Evaluations: Goals and Measures

Evaluation is a systematic approach to assessing how well a system achie...

FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks

Federated Learning (FL) is a distributed learning paradigm that can lear...

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

Hyperparameter optimization (HPO) is crucial for machine learning algori...

FedEval: A Benchmark System with a Comprehensive Evaluation Model for Federated Learning

As an innovative solution for privacy-preserving machine learning (ML), ...

FLGo: A Fully Customizable Federated Learning Platform

Federated learning (FL) has found numerous applications in healthcare, f...

A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design

Federated Learning (FL) has been an area of active research in recent ye...

Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity

Federated learning (FL) is an effective mechanism for data privacy in re...

Please sign up or login with your details

Forgot password? Click here to reset