Controlled Random Search Improves Sample Mining and Hyper-Parameter Optimization

by   Gowtham Muniraju, et al.

A common challenge in machine learning and related fields is the need to efficiently explore high dimensional parameter spaces using small numbers of samples. Typical examples are hyper-parameter optimization in deep learning and sample mining in predictive modeling tasks. All such problems trade-off exploration, which samples the space without knowledge of the target function, and exploitation where information from previous evaluations is used in an adaptive feedback loop. Much of the recent focus has been on the exploitation while exploration is done with simple designs such as Latin hypercube or even uniform random sampling. In this paper, we introduce optimal space-filling sample designs for effective exploration of high dimensional spaces. Specifically, we propose a new parameterized family of sample designs called space-filling spectral designs, and introduce a framework to choose optimal designs for a given sample size and dimension. Furthermore, we present an efficient algorithm to synthesize a given spectral design. Finally, we evaluate the performance of spectral designs in both data space and model space applications. The data space exploration is targeted at recovering complex regression functions in high dimensional spaces. The model space exploration focuses on selecting hyper-parameters for a given neural network architecture. Our empirical studies demonstrate that the proposed approach consistently outperforms state-of-the-art techniques, particularly with smaller design sizes.


page 1

page 2

page 3

page 4


SF-SFD: Stochastic Optimization of Fourier Coefficients to Generate Space-Filling Designs

Due to the curse of dimensionality, it is often prohibitively expensive ...

A Spectral Approach for the Design of Experiments: Design, Analysis and Algorithms

This paper proposes a new approach to construct high quality space-filli...

Distance-distributed design for Gaussian process surrogates

A common challenge in computer experiments and related fields is to effi...

One-Shot Decision-Making with and without Surrogates

One-shot decision making is required in situations in which we can evalu...

Targeted Adaptive Design

Modern advanced manufacturing and advanced materials design often requir...

A Look at the Effect of Sample Design on Generalization through the Lens of Spectral Analysis

This paper provides a general framework to study the effect of sampling ...

A Hierarchical Approach to Scaling Batch Active Search Over Structured Data

Active search is the process of identifying high-value data points in a ...

Please sign up or login with your details

Forgot password? Click here to reset