Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition

by   Saurav Sharma, et al.
Université de Strasbourg

One of the recent advances in surgical AI is the recognition of surgical activities as triplets of (instrument, verb, target). Albeit providing detailed information for computer-assisted intervention, current triplet recognition approaches rely only on single frame features. Exploiting the temporal cues from earlier frames would improve the recognition of surgical action triplets from videos. In this paper, we propose Rendezvous in Time (RiT) - a deep learning model that extends the state-of-the-art model, Rendezvous, with temporal modeling. Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition. We validate our proposal on the challenging surgical triplet dataset, CholecT45, demonstrating an improved recognition of the verb and triplet along with other interactions involving the verb such as (instrument, verb). Qualitative results show that the RiT produces smoother predictions for most triplet instances than the state-of-the-arts. We present a novel attention-based approach that leverages the temporal fusion of video frames to model the evolution of surgical actions and exploit their benefits for surgical triplet recognition.


page 3

page 7


Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Surgical action triplets describe instrument-tissue interactions as (ins...

Why Deep Surgical Models Fail?: Revisiting Surgical Action Triplet Recognition through the Lens of Robustness

Surgical action triplet recognition provides a better understanding of t...

Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos

Out of all existing frameworks for surgical workflow analysis in endosco...

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Recognition of surgical activity is an essential component to develop co...

Shifted-Windows Transformers for the Detection of Cerebral Aneurysms in Microsurgery

Purpose: Microsurgical Aneurysm Clipping Surgery (MACS) carries a high r...

Concept Graph Neural Networks for Surgical Video Understanding

We constantly integrate our knowledge and understanding of the world to ...

Please sign up or login with your details

Forgot password? Click here to reset