Single-DARTS: Towards Stable Architecture Search

08/18/2021
by   Pengfei Hou, et al.
0

Differentiable architecture search (DARTS) marks a milestone in Neural Architecture Search (NAS), boasting simplicity and small search costs. However, DARTS still suffers from frequent performance collapse, which happens when some operations, such as skip connections, zeroes and poolings, dominate the architecture. In this paper, we are the first to point out that the phenomenon is attributed to bi-level optimization. We propose Single-DARTS which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. Even single-level optimization has been previously attempted, no literature provides a systematic explanation on this essential point. Replacing the bi-level optimization, Single-DARTS obviously alleviates performance collapse as well as enhances the stability of architecture search. Experiment results show that Single-DARTS achieves state-of-the-art performance on mainstream search spaces. For instance, on NAS-Benchmark-201, the searched architectures are nearly optimal ones. We also validate that the single-level optimization framework is much more stable than the bi-level one. We hope that this simple yet effective method will give some insights on differential architecture search. The code is available at https://github.com/PencilAndBike/Single-DARTS.git.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/11/2023

Operation-level Progressive Differentiable Architecture Search

Differentiable Neural Architecture Search (DARTS) is becoming more and m...
research
02/12/2020

Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

Differentiable architecture search (DARTS) is a prevailing NAS solution ...
research
04/29/2019

Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Recently, differentiable search methods have made major progress in redu...
research
07/13/2022

MRF-UNets: Searching UNet with Markov Random Fields

UNet [27] is widely used in semantic segmentation due to its simplicity ...
research
11/27/2019

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Differential Architecture Search (DARTS) is now a widely disseminated we...
research
09/20/2019

Understanding and Robustifying Differentiable Architecture Search

Differentiable Architecture Search (DARTS) has attracted a lot of attent...
research
01/16/2023

β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search

Neural Architecture Search has attracted increasing attention in recent ...

Please sign up or login with your details

Forgot password? Click here to reset