One-Shot Neural Architecture Search via Self-Evaluated Template Network

by   Xuanyi Dong, et al.

Neural architecture search (NAS) aims to automate the search procedure of architecture instead of manual design. Even if recent NAS approaches finish the search within days, lengthy training is still required for a specific architecture candidate to get the parameters for its accurate evaluation. Recently one-shot NAS methods are proposed to largely squeeze the tedious training process by sharing parameters across candidates. In this way, the parameters for each candidate can be directly extracted from the shared parameters instead of training them from scratch. However, they have no sense of which candidate will perform better until evaluation so that the candidates to evaluate are randomly sampled and the top-1 candidate is considered the best. In this paper, we propose a Self-Evaluated Template Network (SETN) to improve the quality of the architecture candidates for evaluation so that it is more likely to cover competitive candidates. SETN consists of two components: (1) an evaluator, which learns to indicate the probability of each individual architecture being likely to have a lower validation loss. The candidates for evaluation can thus be selectively sampled according to this evaluator. (2) a template network, which shares parameters among all candidates to amortize the training cost of generated candidates. In experiments, the architecture found by SETN achieves state-of-the-art performance on CIFAR and ImageNet benchmarks within comparable computation costs. Code is publicly available on GitHub:


page 1

page 2

page 3

page 4


PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search

Neural architecture search (NAS) aims to automate architecture engineeri...

Prior-Guided One-shot Neural Architecture Search

Neural architecture search methods seek optimal candidates with efficien...

Blockwisely Supervised Neural Architecture Search with Knowledge Distillation

Neural Architecture Search (NAS), aiming at automatically designing netw...

Can GPT-4 Perform Neural Architecture Search?

We investigate the potential of GPT-4~\cite{gpt4} to perform Neural Arch...

EcoNAS: Finding Proxies for Economical Neural Architecture Search

Neural Architecture Search (NAS) achieves significant progress in many c...

KNAS: Green Neural Architecture Search

Many existing neural architecture search (NAS) solutions rely on downstr...

NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size

Neural architecture search (NAS) has attracted a lot of attention and ha...

Please sign up or login with your details

Forgot password? Click here to reset