SIESTA: Efficient Online Continual Learning with Sleep

by   Md Yousuf Harun, et al.

In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we do not propose a novel method. Instead, we present SIESTA, an incremental improvement to the continual learning algorithm REMIND. Unlike REMIND, SIESTA uses a wake/sleep framework for training, which is well aligned to the needs of on-device learning. SIESTA is far more computationally efficient than existing methods, enabling continual learning on ImageNet-1K in under 3 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications.


Overcoming the Stability Gap in Continual Learning

In many real-world applications, deep neural networks are retrained from...

One Pass ImageNet

We present the One Pass ImageNet (OPIN) problem, which aims to study the...

Avalanche: A PyTorch Library for Deep Continual Learning

Continual learning is the problem of learning from a nonstationary strea...

Mixture of basis for interpretable continual learning with distribution shifts

Continual learning in environments with shifting data distributions is a...

Tackling Online One-Class Incremental Learning by Removing Negative Contrasts

Recent work studies the supervised online continual learning setting whe...

Coresets via Bilevel Optimization for Continual Learning and Streaming

Coresets are small data summaries that are sufficient for model training...

Please sign up or login with your details

Forgot password? Click here to reset