Low-Rank Training of Deep Neural Networks for Emerging Memory Technology

by   Albert Gural, et al.

The recent success of neural networks for solving difficult decision talrt has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. In this work, we address two key challenges for training on edge devices with non-volatile memory: low write density and low auxiliary memory. We present a low-rank training scheme that addresses these challenges while maintaining computational efficiency. We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight writes.


page 16

page 17


Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Network

Training machine learning models on mobile devices has the potential of ...

Exploring Low Rank Training of Deep Neural Networks

Training deep neural networks in low rank, i.e. with factorised layers, ...

Moccasin: Efficient Tensor Rematerialization for Neural Networks

The deployment and training of neural networks on edge computing devices...

A Sequential Concept Drift Detection Method for On-Device Learning on Low-End Edge Devices

A practical issue of edge AI systems is that data distributions of train...

Counting Cards: Exploiting Weight and Variance Distributions for Robust Compute In-Memory

Compute in-memory (CIM) is a promising technique that minimizes data tra...

Lazy Evaluation of Convolutional Filters

In this paper we propose a technique which avoids the evaluation of cert...

Network insensitivity to parameter noise via adversarial regularization

Neuromorphic neural network processors, in the form of compute-in-memory...

Please sign up or login with your details

Forgot password? Click here to reset