A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation

by   Mitsuru Kusumoto, et al.
Preferred Infrastructure
The University of Tokyo

Recomputation algorithms collectively refer to a family of methods that aims to reduce the memory consumption of the backpropagation by selectively discarding the intermediate results of the forward propagation and recomputing the discarded results as needed. In this paper, we will propose a novel and efficient recomputation method that can be applied to a wider range of neural nets than previous methods. We use the language of graph theory to formalize the general recomputation problem of minimizing the computational overhead under a fixed memory budget constraint, and provide a dynamic programming solution to the problem. Our method can reduce the peak memory consumption on various benchmark networks by 36 by other methods.


page 1

page 2

page 3

page 4


Memory-Efficient Backpropagation Through Time

We propose a novel approach to reduce memory consumption of the backprop...

Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices

Recent advances demonstrate that irregularly wired neural networks from ...

Backpropagation for long sequences: beyond memory constraints with constant overheads

Naive backpropagation through time has a memory footprint that grows lin...

Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models

We propose a memory efficient method, named Stochastic Backpropagation (...

TinyProp – Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning

Training deep neural networks using backpropagation is very memory and c...

Lossy Checkpoint Compression in Full Waveform Inversion

This paper proposes a new method that combines check-pointing methods wi...

AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models

Existing customization methods require access to multiple reference exam...

Please sign up or login with your details

Forgot password? Click here to reset