Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation

by   Mengya Xu, et al.
National University of Singapore

Generating surgical reports aimed at surgical scene understanding in robot-assisted surgery can contribute to documenting entry tasks and post-operative analysis. Despite the impressive outcome, the deep learning model degrades the performance when applied to different domains encountering domain shifts. In addition, there are new instruments and variations in surgical tissues appeared in robotic surgery. In this work, we propose class-incremental domain adaptation (CIDA) with a multi-layer transformer-based model to tackle the new classes and domain shift in the target domain to generate surgical reports during robotic surgery. To adapt incremental classes and extract domain invariant features, a class-incremental (CI) learning method with supervised contrastive (SupCon) loss is incorporated with a feature extractor. To generate caption from the extracted feature, curriculum by one-dimensional gaussian smoothing (CBS) is integrated with a multi-layer transformer-based caption prediction model. CBS smoothes the features embedding using anti-aliasing and helps the model to learn domain invariant features. We also adopt label smoothing (LS) to calibrate prediction probability and obtain better feature representation with both feature extractor and captioning model. The proposed techniques are empirically evaluated by using the datasets of two surgical domains, such as nephrectomy operations and transoral robotic surgery. We observe that domain invariant feature learning and the well-calibrated network improves the surgical report generation performance in both source and target domain under domain shift and unseen classes in the manners of one-shot and few-shot learning. The code is publicly available at


page 1

page 2

page 3

page 4


Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery

Generating a surgical report in robot-assisted surgery, in the form of n...

Rethinking Surgical Captioning: End-to-End Window-Based MLP Transformer Using Patches

Surgical captioning plays an important role in surgical instruction pred...

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Learning to infer graph representations and performing spatial reasoning...

Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

For robot-assisted surgery, an accurate surgical report reflects clinica...

Joint Domain Alignment and Discriminative Feature Learning for Unsupervised Deep Domain Adaptation

Recently, considerable effort has been devoted to deep domain adaptation...

Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge

Surgical data science is revolutionizing minimally invasive surgery by e...

Please sign up or login with your details

Forgot password? Click here to reset