Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos

by   Chinedu Innocent Nwoye, et al.
Université de Strasbourg

Purpose: Real-time surgical tool tracking is a core component of the future intelligent operating room (OR), because it is highly instrumental to analyze and understand the surgical activities. Current methods for surgical tool tracking in videos need to be trained on data in which the spatial position of the tools is manually annotated. Generating such training data is difficult and time-consuming. Instead, we propose to use solely binary presence annotations to train a tool tracker for laparoscopic videos. Methods: The proposed approach is composed of a CNN + Convolutional LSTM (ConvLSTM) neural network trained end-to-end, but weakly supervised on tool binary presence labels only. We use the ConvLSTM to model the temporal dependencies in the motion of the surgical tools and leverage its spatio-temporal ability to smooth the class peak activations in the localization heat maps (Lh-maps). Results: We build a baseline tracker on top of the CNN model and demonstrate that our approach based on the ConvLSTM outperforms the baseline in tool presence detection, spatial localization, and motion tracking by over 5.0 13.9 Conclusions: In this paper, we demonstrate that binary presence labels are sufficient for training a deep learning tracking model using our proposed method. We also show that the ConvLSTM can leverage the spatio-temporal coherence of consecutive image frames across a surgical video to improve tool presence detection, spatial localization, and motion tracking.


Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos

Surgical tool localization is an essential task for the automatic analys...

Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks

Five billion people in the world lack access to quality surgical care. S...

Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery

Open, or non-laparoscopic surgery, represents the vast majority of all o...

TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery

Purpose: Tracking the 3D motion of the surgical tool and the patient ana...

Surgical tool classification and localization: results and methods from the MICCAI 2022 SurgToolLoc challenge

The ability to automatically detect and track surgical instruments in en...

Please sign up or login with your details

Forgot password? Click here to reset