DeepAI AI Chat
Log In Sign Up

DARTS without a Validation Set: Optimizing the Marginal Likelihood

by   Miroslav Fil, et al.
University of Oxford

The success of neural architecture search (NAS) has historically been limited by excessive compute requirements. While modern weight-sharing NAS methods such as DARTS are able to finish the search in single-digit GPU days, extracting the final best architecture from the shared weights is notoriously unreliable. Training-Speed-Estimate (TSE), a recently developed generalization estimator with a Bayesian marginal likelihood interpretation, has previously been used in place of the validation loss for gradient-based optimization in DARTS. This prevents the DARTS skip connection collapse, which significantly improves performance on NASBench-201 and the original DARTS search space. We extend those results by applying various DARTS diagnostics and show several unusual behaviors arising from not using a validation set. Furthermore, our experiments yield concrete examples of the depth gap and topology selection in DARTS having a strongly negative impact on the search performance despite generally receiving limited attention in the literature compared to the operations selection.


page 1

page 2

page 3

page 4


Rethinking Architecture Selection in Differentiable NAS

Differentiable Neural Architecture Search is one of the most popular Neu...

Zero-Cost Proxies Meet Differentiable Architecture Search

Differentiable neural architecture search (NAS) has attracted significan...

Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search

Reliable yet efficient evaluation of generalisation performance of a pro...

Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search

High sensitivity of neural architecture search (NAS) methods against the...

Improving One-shot NAS by Suppressing the Posterior Fading

There is a growing interest in automated neural architecture search (NAS...

Backpropagation-Free 4D Continuous Ant-Based Neural Topology Search

Continuous Ant-based Topology Search (CANTS) is a previously introduced ...

Bayesian Model Selection, the Marginal Likelihood, and Generalization

How do we compare between hypotheses that are entirely consistent with o...