DeepAI AI Chat
Log In Sign Up

Pre-training helps Bayesian optimization too

by   Zi Wang, et al.

Bayesian optimization (BO) has become a popular strategy for global optimization of many expensive real-world functions. Contrary to a common belief that BO is suited to optimizing black-box functions, it actually requires domain knowledge on characteristics of those functions to deploy BO successfully. Such domain knowledge often manifests in Gaussian process priors that specify initial beliefs on functions. However, even with expert knowledge, it is not an easy task to select a prior. This is especially true for hyperparameter tuning problems on complex machine learning models, where landscapes of tuning objectives are often difficult to comprehend. We seek an alternative practice for setting these functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods.


page 1

page 2

page 3

page 4


Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers

The performance of deep neural networks can be highly sensitive to the c...

HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processes

Bayesian optimization (BO), while proved highly effective for many black...

Prior-guided Bayesian Optimization

While Bayesian Optimization (BO) is a very popular method for optimizing...

SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees

Gradient boosted trees (GBTs) are ubiquitous models used by researchers,...

Amazon SageMaker Automatic Model Tuning: Scalable Black-box Optimization

Tuning complex machine learning systems is challenging. Machine learning...

Hyperparameter Transfer Learning with Adaptive Complexity

Bayesian optimization (BO) is a sample efficient approach to automatical...

Practical Phase Retrieval Using Double Deep Image Priors

Phase retrieval (PR) concerns the recovery of complex phases from comple...