Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections

by   Xichuan Zhou, et al.

Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data. However, implementing a deep neural network in embedded systems is a challenging task, because a typical deep neural network, such as a Deep Belief Network using 128x128 images as input, could exhaust Giga bytes of memory and result in bandwidth and computing bottleneck. To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections. The proposed method adaptively reduces the weights associated with negligible features to zero, leading to sparse feedforward network architecture. Furthermore, since the small proportion of important weights are significantly larger than zero, they can be robustly thresholded and represented using single-bit integers (-1 and +1), leading to implementations of deep neural networks with sparse and binary connections. Our experiments showed that, for the application of recognizing MNIST handwritten digits, the features extracted by a two-layer Deep Adaptive Network with about 25 classification accuracy, which was almost the same with the standard Deep Belief Network (97.3 the sparse-and-binary-weighted deep neural network could save about 99.3 memory and 99.9 accuracy for pattern recognition applications.


HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

Binary Neural Networks (BNNs), neural networks with weights and activati...

At-Scale Sparse Deep Neural Network Inference with Efficient GPU Implementation

This paper presents GPU performance optimization and scaling results for...

AdaDNNs: Adaptive Ensemble of Deep Neural Networks for Scene Text Recognition

Recognizing text in the wild is a really challenging task because of com...

Cramnet: Layer-wise Deep Neural Network Compression with Knowledge Transfer from a Teacher Network

Neural Networks accomplish amazing things, but they suffer from computat...

Image classification using quantum inference on the D-Wave 2X

We use a quantum annealing D-Wave 2X computer to obtain solutions to NP-...

Increasing the Accuracy of a Neural Network Using Frequency Selective Mesh-to-Grid Resampling

Neural networks are widely used for almost any task of recognizing image...

Please sign up or login with your details

Forgot password? Click here to reset