Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery

by   Mengya Xu, et al.

Generating a surgical report in robot-assisted surgery, in the form of natural language expression of surgical scene understanding, can play a significant role in document entry tasks, surgical training, and post-operative analysis. Despite the state-of-the-art accuracy of the deep learning algorithm, the deployment performance often drops when applied to the Target Domain (TD) data. For this purpose, we develop a multi-layer transformer-based model with the gradient reversal adversarial learning to generate a caption for the multi-domain surgical images that can describe the semantic relationship between instruments and surgical Region of Interest (ROI). In the gradient reversal adversarial learning scheme, the gradient multiplies with a negative constant and updates adversarially in backward propagation, discriminating between the source and target domains and emerging domain-invariant features. We also investigate model calibration with label smoothing technique and the effect of a well-calibrated model for the penultimate layer's feature representation and Domain Adaptation (DA). We annotate two robotic surgery datasets of MICCAI robotic scene segmentation and Transoral Robotic Surgery (TORS) with the captions of procedures and empirically show that our proposed method improves the performance in both source and target domain surgical reports generation in the manners of unsupervised, zero-shot, one-shot, and few-shot learning.


page 1

page 2

page 4

page 6


Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation

Generating surgical reports aimed at surgical scene understanding in rob...

Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

For robot-assisted surgery, an accurate surgical report reflects clinica...

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Learning to infer graph representations and performing spatial reasoning...

Improving rigid 3D calibration for robotic surgery

Autonomy is the frontier of research in robotic surgery and its aim is t...

Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning

Robot-assisted surgery is an emerging technology which has undergone rap...

Mixed Reality using Illumination-aware Gradient Mixing in Surgical Telepresence: Enhanced Multi-layer Visualization

Background and aim: Surgical telepresence using augmented perception has...

Please sign up or login with your details

Forgot password? Click here to reset