Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay

by   Guankun Wang, et al.
The University of Sydney
The Chinese University of Hong Kong

Endoscopy is a widely used technique for the early detection of diseases or robotic-assisted minimally invasive surgery (RMIS). Numerous deep learning (DL)-based research works have been developed for automated diagnosis or processing of endoscopic view. However, existing DL models may suffer from catastrophic forgetting. When new target classes are introduced over time or cross institutions, the performance of old classes may suffer severe degradation. More seriously, data privacy and storage issues may lead to the unavailability of old data when updating the model. Therefore, it is necessary to develop a continual learning (CL) methodology to solve the problem of catastrophic forgetting in endoscopic image segmentation. To tackle this, we propose a Endoscopy Continual Semantic Segmentation (EndoCSS) framework that does not involve the storage and privacy issues of exemplar data. The framework includes a mini-batch pseudo-replay (MB-PR) mechanism and a self-adaptive noisy cross-entropy (SAN-CE) loss. The MB-PR strategy circumvents privacy and storage issues by generating pseudo-replay images through a generative model. Meanwhile, the MB-PR strategy can also correct the model deviation to the replay data and current training data, which is aroused by the significant difference in the amount of current and replay images. Therefore, the model can perform effective representation learning on both new and old tasks. SAN-CE loss can help model fitting by adjusting the model's output logits, and also improve the robustness of training. Extensive continual semantic segmentation (CSS) experiments on public datasets demonstrate that our method can robustly and effectively address the catastrophic forgetting brought by class increment in endoscopy scenes. The results show that our framework holds excellent potential for real-world deployment in a streaming learning manner.


page 2

page 4

page 8


PLOP: Learning without Forgetting for Continual Semantic Segmentation

Deep learning approaches are nowadays ubiquitously used to tackle comput...

Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation

Deep learning approaches are nowadays ubiquitously used to tackle comput...

RECALL: Replay-based Continual Learning in Semantic Segmentation

Deep networks allow to obtain outstanding results in semantic segmentati...

DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation

The Class Incremental Semantic Segmentation (CISS) extends the tradition...

Continual Semantic Segmentation with Automatic Memory Sample Selection

Continual Semantic Segmentation (CSS) extends static semantic segmentati...

CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation

Operating a robot in the open world requires a high level of robustness ...

Please sign up or login with your details

Forgot password? Click here to reset