DeepAI AI Chat
Log In Sign Up

Diversity Networks: Neural Network Compression Using Determinantal Point Processes

by   Zelda Mariet, et al.

We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.


Towards thinner convolutional neural networks through Gradually Global Pruning

Deep network pruning is an effective method to reduce the storage and co...

Neural networks embrace learned diversity

Diversity conveys advantages in nature, yet homogeneous neurons typicall...

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

State-of-the-art neural networks are getting deeper and wider. While the...

Syntactic vs Semantic Linear Abstraction and Refinement of Neural Networks

Abstraction is a key verification technique to improve scalability. Howe...

Within-layer Diversity Reduces Generalization Gap

Neural networks are composed of multiple layers arranged in a hierarchic...

Compressing Neural Networks using the Variational Information Bottleneck

Neural networks can be compressed to reduce memory and computational req...

WLD-Reg: A Data-dependent Within-layer Diversity Regularizer

Neural networks are composed of multiple layers arranged in a hierarchic...