Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets

by   Chinedu Innocent Nwoye, et al.
Université de Strasbourg

In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.


CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

Context-aware decision support in the operating room can foster surgical...

Surgical Phase and Instrument Recognition: How to identify appropriate Dataset Splits

Purpose: The development of machine learning models for surgical workflo...

Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Surgical action triplets describe instrument-tissue interactions as (ins...

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for b...

DC-BENCH: Dataset Condensation Benchmark

Dataset Condensation is a newly emerging technique aiming at learning a ...

Wizard of Search Engine: Access to Information Through Conversations with Search Engines

Conversational information seeking (CIS) is playing an increasingly impo...

Code Repositories


Laparoscopic video dataset for surgical action triplet recognition

view repo


A transformer-inspired neural network for surgical action triplet recognition from laparoscopic videos.

view repo

Please sign up or login with your details

Forgot password? Click here to reset