Fully Parallel Hyperparameter Search: Reshaped Space-Filling

by   M. -L. Cauwet, et al.

Space-filling designs such as scrambled-Hammersley, Latin Hypercube Sampling and Jittered Sampling have been proposed for fully parallel hyperparameter search, and were shown to be more effective than random or grid search. In this paper, we show that these designs only improve over random search by a constant factor. In contrast, we introduce a new approach based on reshaping the search distribution, which leads to substantial gains over random search, both theoretically and empirically. We propose two flavors of reshaping. First, when the distribution of the optimum is some known P_0, we propose Recentering, which uses as search distribution a modified version of P_0 tightened closer to the center of the domain, in a dimension-dependent and budget-dependent manner. Second, we show that in a wide range of experiments with P_0 unknown, using a proposed Cauchy transformation, which simultaneously has a heavier tail (for unbounded hyperparameters) and is closer to the boundaries (for bounded hyperparameters), leads to improved performances. Besides artificial experiments and simple real world tests on clustering or Salmon mappings, we check our proposed methods on expensive artificial intelligence tasks such as attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks.


page 1

page 2

page 3

page 4


Random Search for Hyperparameters using Determinantal Point Processes

We propose the use of k-determinantal point processes in hyperparameter ...

A Population-based Hybrid Approach to Hyperparameter Optimization for Neural Networks

In recent years, large amounts of data have been generated, and computer...

Hyperparameter optimization of orthogonal functions in the numerical solution of differential equations

This paper considers the hyperparameter optimization problem of mathemat...

Variance Reduction for Better Sampling in Continuous Domains

Design of experiments, random search, initialization of population-based...

PHS: A Toolbox for Parallel Hyperparameter Search

We introduce an open source python framework named PHS - Parallel Hyperp...

Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space

Hyperparameter optimization is both a practical issue and an interesting...

How to scale hyperparameters for quickshift image segmentation

Quickshift is a popular algorithm for image segmentation, used as a prep...

Please sign up or login with your details

Forgot password? Click here to reset