Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

by   Mobarakol Islam, et al.
National University of Singapore

Learning to infer graph representations and performing spatial reasoning in a complex surgical environment can play a vital role in surgical scene understanding in robotic surgery. For this purpose, we develop an approach to generate the scene graph and predict surgical interactions between instruments and surgical region of interest (ROI) during robot-assisted surgery. We design an attention link function and integrate with a graph parsing network to recognize the surgical interactions. To embed each node with corresponding neighbouring node features, we further incorporate SageConv into the network. The scene graph generation and active edge classification mostly depend on the embedding or feature extraction of node and edge features from complex image representation. Here, we empirically demonstrate the feature extraction methods by employing label smoothing weighted loss. Smoothing the hard label can avoid the over-confident prediction of the model and enhances the feature representation learned by the penultimate layer. To obtain the graph scene label, we annotate the bounding box and the instrument-ROI interactions on the robotic scene segmentation challenge 2018 dataset with an experienced clinical expert in robotic surgery and employ it to evaluate our propositions.


Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery

Generating a surgical report in robot-assisted surgery, in the form of n...

Towards Graph Representation Learning Based Surgical Workflow Anticipation

Surgical workflow anticipation can give predictions on what steps to con...

Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

For robot-assisted surgery, an accurate surgical report reflects clinica...

4D-OR: Semantic Scene Graphs for OR Domain Modeling

Surgical procedures are conducted in highly complex operating rooms (OR)...

Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation

Generating surgical reports aimed at surgical scene understanding in rob...

Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

Global and local relational reasoning enable scene understanding models ...

ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Real-time tool segmentation from endoscopic videos is an essential part ...

Please sign up or login with your details

Forgot password? Click here to reset