Learning Features with Parameter-Free Layers

by   Dongyoon Han, et al.

Trainable layers such as convolutional building blocks are the standard network design choices by learning parameters to capture the global context through successive spatial operations. When designing an efficient network, trainable layers such as the depthwise convolution is the source of efficiency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice. This paper argues that simple built-in parameter-free operations can be a favorable alternative to the efficient trainable layers replacing spatial operations in a network architecture. We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers. Extensive experimental analyses based on layer-level studies with fully-trained models and neural architecture searches are provided to investigate whether parameter-free operations such as the max-pool are functional. The studies eventually give us a simple yet effective idea for redesigning network architectures, where the parameter-free operations are heavily used as the main building block without sacrificing the model accuracy as much. Experimental results on the ImageNet dataset demonstrate that the network architectures with parameter-free operations could enjoy the advantages of further efficiency in terms of model speed, the number of the parameters, and FLOPs. Code and ImageNet pretrained models are available at https://github.com/naver-ai/PfLayer.


page 14

page 16

page 17

page 18


Speeding Up EfficientNet: Selecting Update Blocks of Convolutional Neural Networks using Genetic Algorithm in Transfer Learning

The performance of convolutional neural networks (CNN) depends heavily o...

SplitMixer: Fat Trimmed From MLP-like Models

We present SplitMixer, a simple and lightweight isotropic MLP-like archi...

Orthogonal Transform Domain Approaches for the Convolutional Layer

In this paper, we propose a set of transform-based neural network layers...

Efficient Image Super-Resolution Using Pixel Attention

This work aims at designing a lightweight convolutional neural network f...

Fast Sparse ConvNets

Historically, the pursuit of efficient inference has been one of the dri...

Adaptable Adapters

State-of-the-art pretrained NLP models contain a hundred million to tril...

NoMorelization: Building Normalizer-Free Models from a Sample's Perspective

The normalizing layer has become one of the basic configurations of deep...

Please sign up or login with your details

Forgot password? Click here to reset