DHP: Differentiable Meta Pruning via HyperNetworks

by   Yawei Li, et al.

Network pruning has been the driving force for the efficient inference of neural networks and the alleviation of model storage and transmission burden. Traditional network pruning methods focus on the per-filter influence on the network accuracy by analyzing the filter distribution. With the advent of AutoML and neural architecture search (NAS), pruning has become topical with automatic mechanism and searching based architecture optimization. However, current automatic designs rely on either reinforcement learning or evolutionary algorithm, which often do not have a theoretical convergence guarantee or do not converge in a meaningful time limit. In this paper, we propose a differentiable pruning method via hypernetworks for automatic network pruning and layer-wise configuration optimization. A hypernetwork is designed to generate the weights of the backbone network. The input of the hypernetwork, namely, the latent vectors control the output channels of the layers of backbone network. By applying ℓ_1 sparsity regularization to the latent vectors and utilizing proximal gradient, sparse latent vectors can be obtained with removed zero elements. Thus, the corresponding elements of the hypernetwork outputs can also be removed, achieving the effect of network pruning. The latent vectors of all the layers are pruned together, resulting in an automatic layer configuration. Extensive experiments are conducted on various networks for image classification, single image super-resolution, and denoising. And the experimental results validate the proposed method.


DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation

Budgeted pruning is the problem of pruning under resource constraints. I...

Tiered Pruning for Efficient Differentialble Inference-Aware Neural Architecture Search

We propose three novel pruning techniques to improve the cost and result...

Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization

Neural architecture search (NAS) and network pruning are widely studied ...

Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution

This paper proposes a trilevel neural architecture search (NAS) method f...

Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks

Recently, deep learning has become a de facto standard in machine learni...

Neural Network Panning: Screening the Optimal Sparse Network Before Training

Pruning on neural networks before training not only compresses the origi...

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization

Neural networks have made great progress in pixel to pixel image process...

Code Repositories


This is the official implementation of "DHP: Differentiable Meta Pruning via HyperNetworks".

view repo

Please sign up or login with your details

Forgot password? Click here to reset