DeepAI AI Chat
Log In Sign Up

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

by   Zhen Wang, et al.
ETH Zurich
Alibaba Group

Hyperparameter optimization (HPO) is crucial for machine learning algorithms to achieve satisfactory performance, whose progress has been boosted by related benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for traditional centralized learning while ignoring federated learning (FL), a promising paradigm for collaboratively learning models from dispersed data. In this paper, we first identify some uniqueness of HPO for FL algorithms from various aspects. Due to this uniqueness, existing HPO benchmarks no longer satisfy the need to compare HPO methods in the FL setting. To facilitate the research of HPO in the FL setting, we propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods. We open-source FedHPO-B at


page 1

page 2

page 3

page 4


FedScale: Benchmarking Model and System Performance of Federated Learning at Scale

We present FedScale, a diverse set of challenging and realistic benchmar...

UniFed: A Benchmark for Federated Learning Frameworks

Federated Learning (FL) has become a practical and popular paradigm in m...

FederatedScope: A Comprehensive and Flexible Federated Learning Platform via Message Passing

Although remarkable progress has been made by the existing federated lea...

An Isolated Data Island Benchmark Suite for Federated Learning

Federated learning (FL) is a new machine learning paradigm, the goal of ...

SLPerf: a Unified Framework for Benchmarking Split Learning

Data privacy concerns has made centralized training of data, which is sc...

YAHPO Gym – Design Criteria and a new Multifidelity Benchmark for Hyperparameter Optimization

When developing and analyzing new hyperparameter optimization (HPO) meth...

A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design

Federated Learning (FL) has been an area of active research in recent ye...