Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

by   Chinedu Innocent Nwoye, et al.
Université de Strasbourg

Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Activation Guide (CAG), which uses the instrument activation maps to guide the verb and target recognition. To model the recognition of multiple triplets in the same frame, we also propose a trainable 3D Interaction Space, which captures the associations between the triplet components. Finally, we demonstrate the significance of these contributions via several ablation studies and comparisons to baselines on CholecT40.


page 2

page 9

page 13


Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos

Out of all existing frameworks for surgical workflow analysis in endosco...

CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

Context-aware decision support in the operating room can foster surgical...

Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Surgical action triplets describe instrument-tissue interactions as (ins...

Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition

One of the recent advances in surgical AI is the recognition of surgical...

Analyzing Before Solving: Which Parameters Influence Low-Level Surgical Activity Recognition

Automatic low-level surgical activity recognition is today well-known te...

A Context-Aware Loss Function for Action Spotting in Soccer Videos

Action spotting is an important element of general activity understandin...

Please sign up or login with your details

Forgot password? Click here to reset