Non-deep Networks

by   Ankit Goyal, et al.

Depth is the hallmark of deep neural networks. But more depth means more sequential computation and higher latency. This begs the question – is it possible to build high-performing "non-deep" neural networks? We show that it is. To do so, we use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, we show, for the first time, that a network with a depth of just 12 can achieve top-1 accuracy over 80 ImageNet, 96 a low-depth (12) backbone can achieve an AP of 48 scaling rules for our design and show how to increase performance without changing the network's depth. Finally, we provide a proof of concept for how non-deep networks could be used to build low-latency recognition systems. Code is available at


page 1

page 2

page 3

page 4


Stabilizing Equilibrium Models by Jacobian Regularization

Deep equilibrium networks (DEQs) are a new class of models that eschews ...

Layer Folding: Neural Network Depth Reduction using Activation Linearization

Despite the increasing prevalence of deep neural networks, their applica...

Building a Regular Decision Boundary with Deep Networks

In this work, we build a generic architecture of Convolutional Neural Ne...

Latency-aware Unified Dynamic Networks for Efficient Image Recognition

Dynamic computation has emerged as a promising avenue to enhance the inf...

Towards Interpretable Deep Networks for Monocular Depth Estimation

Deep networks for Monocular Depth Estimation (MDE) have achieved promisi...

Designing Deep Networks for Scene Recognition

Most deep learning backbones are evaluated on ImageNet. Using scenery im...

Low Latency Privacy Preserving Inference

When applying machine learning to sensitive data, one has to balance bet...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset