Design Exploration of Hybrid CMOS-OxRAM Deep Generative Architectures

by   Vivek Parmar, et al.

Deep Learning and its applications have gained tremendous interest recently in both academia and industry. Restricted Boltzmann Machines (RBMs) offer a key methodology to implement deep learning paradigms. This paper presents a novel approach for realizing hybrid CMOS-OxRAM based deep generative models (DGM). In our proposed hybrid DGM architectures, HfOx based (filamentary-type switching) OxRAM devices are extensively used for realizing multiple computational and non-computational functions such as: (i) Synapses (weights), (ii) internal neuron-state storage, (iii) stochastic neuron activation and (iv) programmable signal normalization. To validate the proposed scheme we have simulated two different architectures: (i) Deep Belief Network (DBN) and (ii) Stacked Denoising Autoencoder for classification and reconstruction of hand-written digits from a reduced MNIST dataset of 6000 images. Contrastive-divergence (CD) specially optimized for OxRAM devices was used to drive the synaptic weight update mechanism of each layer in the network. Overall learning rule was based on greedy-layer wise learning with no back propagation which allows the network to be trained to a good pre-training stage. Performance of the simulated hybrid CMOS-RRAM DGM model matches closely with software based model for a 2-layers deep network. Top-3 test accuracy achieved by the DBN was 95.5 network was 0.003, lower than software based approach. Endurance analysis of the simulated architectures show that for 200 epochs of training (single RBM layer), maximum switching events/per OxRAM device was 7000 cycles.


Highly-scalable stochastic neuron based on Ovonic Threshold Switch (OTS) and its applications in Restricted Boltzmann Machine (RBM)

Interest in Restricted Boltzmann Machine (RBM) is growing as a generativ...

Layer-wise learning of deep generative models

When using deep, multi-layered architectures to build generative models ...

Training Deep Boltzmann Networks with Sparse Ising Machines

The slowing down of Moore's law has driven the development of unconventi...

Accelerating Deep Learning with Memcomputing

Restricted Boltzmann machines (RBMs) and their extensions, often called ...

Inference with Hybrid Bio-hardware Neural Networks

To understand the learning process in brains, biologically plausible alg...

A developmental approach for training deep belief networks

Deep belief networks (DBNs) are stochastic neural networks that can extr...

Deep learning with t-exponential Bayesian kitchen sinks

Bayesian learning has been recently considered as an effective means of ...

Please sign up or login with your details

Forgot password? Click here to reset