Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

by   Deepak Alapatt, et al.
Université de Strasbourg

A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised framework used for video semantic segmentation of surgical videos. In this work, we show that autoencoder networks can be used to efficiently provide both spatial and temporal supervisory signals to train deep learning models. We test our method on a newly introduced video dataset of laparoscopic cholecystectomy procedures, Endoscapes, and an adaptation of a public dataset of cataract surgeries, CaDIS. We demonstrate that lower-dimensional representations of predicted masks can be leveraged to provide a consistent improvement on both sparsely labeled datasets with no additional computational cost at inference time. Further, the TCNN framework is model-agnostic and can be used in conjunction with other model design choices with minimal additional complexity.


page 1

page 3

page 4

page 6

page 8


A spatio-temporal network for video semantic segmentation in surgical videos

Semantic segmentation in surgical videos has applications in intra-opera...

Improving Semantic Segmentation through Spatio-Temporal Consistency Learned from Videos

We leverage unsupervised learning of depth, egomotion, and camera intrin...

Virtual Temporal Samples for Recurrent Neural Networks: applied to semantic segmentation in agriculture

This paper explores the potential for performing temporal semantic segme...

Coherent Loss: A Generic Framework for Stable Video Segmentation

Video segmentation approaches are of great importance for numerous visio...

CaDSS: Cataract Dataset for Semantic Segmentation

Video signals provide a wealth of information about surgical procedures ...

Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints

Purpose: Vision-based robot tool segmentation plays a fundamental role i...

Future semantic segmentation of time-lapsed videos with large temporal displacement

An important aspect of video understanding is the ability to predict the...

Please sign up or login with your details

Forgot password? Click here to reset