Distilled Replay: Overcoming Forgetting through Synthetic Samples

03/29/2021
by   Andrea Rosasco, et al.
7

Replay strategies are Continual Learning techniques which mitigate catastrophic forgetting by keeping a buffer of patterns from previous experience, which are interleaved with new data during training. The amount of patterns stored in the buffer is a critical parameter which largely influences the final performance and the memory footprint of the approach. This work introduces Distilled Replay, a novel replay strategy for Continual Learning which is able to mitigate forgetting by keeping a very small buffer (up to 1 pattern per class) of highly informative samples. Distilled Replay builds the buffer through a distillation process which compresses a large dataset into a tiny set of informative examples. We show the effectiveness of our Distilled Replay against naive replay, which randomly samples patterns from the dataset, on four popular Continual Learning benchmarks.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/19/2020

Using Hindsight to Anchor Past Knowledge in Continual Learning

In continual learning, the learner faces a stream of data whose distribu...
05/23/2023

Continual Learning with Strong Experience Replay

Continual Learning (CL) aims at incrementally learning new tasks without...
08/03/2023

Improving Replay Sample Selection and Storage for Less Forgetting in Continual Learning

Continual learning seeks to enable deep learners to train on a series of...
02/14/2022

Memory Replay with Data Compression for Continual Learning

Continual learning needs to overcome catastrophic forgetting of the past...
10/12/2020

Rethinking Experience Replay: a Bag of Tricks for Continual Learning

In Continual Learning, a Neural Network is trained on a stream of data w...
10/14/2021

Continual Learning on Noisy Data Streams via Self-Purified Replay

Continually learning in the real world must overcome many challenges, am...
08/21/2021

Principal Gradient Direction and Confidence Reservoir Sampling for Continual Learning

Task-free online continual learning aims to alleviate catastrophic forge...

Code Repositories

DistilledReplay

Code for the pubblication "Distilled Replay: Overcoming Forgetting through Synthetic Examples"


view repo

Please sign up or login with your details

Forgot password? Click here to reset