Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures

by   Sanat Ramesh, et al.
University of Verona
Université de Strasbourg

Purpose: Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Method: We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). Results: We present experimental results from several baseline models for both phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN method outperforms in both phase and step recognition by 1-2 precision and recall, compared to single-task methods. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6 based models in accuracy, precision, and recall. Conclusion: In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on the Bypass40 gastric bypass dataset with multi-level annotations. The proposed method shows that the joint modeling of phases and steps is beneficial to improve the overall recognition of each type of activity.


page 3

page 8

page 10


Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition

Automatic recognition of fine-grained surgical activities, called steps,...

Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks

Modeling and recognition of surgical activities poses an interesting res...

Real-time analysis of cataract surgery videos using statistical models

The automatic analysis of the surgical process, from videos recorded dur...

Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation

Surgical phase recognition (SPR) is a crucial element in the digital tra...

Novel evaluation of surgical activity recognition models using task-based efficiency metrics

Purpose: Surgical task-based metrics (rather than entire procedure metri...

Quantitative Characterization of Components of Computer Assisted Interventions

Purpose: We propose a mathematical framework for quantitative analysis w...

Please sign up or login with your details

Forgot password? Click here to reset