Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets

04/11/2022
by   Chinedu Innocent Nwoye, et al.
Université de Strasbourg
0

In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.

READ FULL TEXT
04/10/2022

CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

Context-aware decision support in the operating room can foster surgical...
06/29/2023

Surgical Phase and Instrument Recognition: How to identify appropriate Dataset Splits

Purpose: The development of machine learning models for surgical workflo...
07/18/2023

Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Surgical action triplets describe instrument-tissue interactions as (ins...
06/10/2021

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for b...
07/20/2022

DC-BENCH: Dataset Condensation Benchmark

Dataset Condensation is a newly emerging technique aiming at learning a ...
05/18/2021

Wizard of Search Engine: Access to Information Through Conversations with Search Engines

Conversational information seeking (CIS) is playing an increasingly impo...

Code Repositories

cholect45

Laparoscopic video dataset for surgical action triplet recognition


view repo

rendezvous

A transformer-inspired neural network for surgical action triplet recognition from laparoscopic videos.


view repo

Please sign up or login with your details

Forgot password? Click here to reset