EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions

by   Laurens Bliek, et al.

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the performance of the six algorithms on all evaluated problem instances. This results in a unique new dataset that lowers the bar for researching new methods as the number of expensive evaluations required for comparison is significantly reduced.


page 1

page 7


Asynchronous ε-Greedy Bayesian Optimisation

Bayesian Optimisation (BO) is a popular surrogate model-based approach f...

Continuous surrogate-based optimization algorithms are well-suited for expensive discrete problems

One method to solve expensive black-box optimization problems is to use ...

Identifying Properties of Real-World Optimisation Problems through a Questionnaire

Optimisation algorithms are commonly compared on benchmarks to get insig...

Meta-Surrogate Benchmarking for Hyperparameter Optimization

Despite the recent progress in hyperparameter optimization (HPO), availa...

MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learning

In Bayesian optimisation, we often seek to minimise the black-box object...

Optimal Observation-Intervention Trade-Off in Optimisation Problems with Causal Structure

We consider the problem of optimising an expensive-to-evaluate grey-box ...

Please sign up or login with your details

Forgot password? Click here to reset