AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery

by   Mobarakol Islam, et al.

Surgical scene understanding and multi-tasking learning are crucial for image-guided robotic surgery. Training a real-time robotic system for the detection and segmentation of high-resolution images provides a challenging problem with the limited computational resource. The perception drawn can be applied in effective real-time feedback, surgical skill assessment, and human-robot collaborative surgeries to enhance surgical outcomes. For this purpose, we develop a novel end-to-end trainable real-time Multi-Task Learning (MTL) model with weight-shared encoder and task-aware detection and segmentation decoders. Optimization of multiple tasks at the same convergence point is vital and presents a complex problem. Thus, we propose an asynchronous task-aware optimization (ATO) technique to calculate task-oriented gradients and train the decoders independently. Moreover, MTL models are always computationally expensive, which hinder real-time applications. To address this challenge, we introduce a global attention dynamic pruning (GADP) by removing less significant and sparse parameters. We further design a skip squeeze and excitation (SE) module, which suppresses weak features, excites significant features and performs dynamic spatial and channel-wise feature re-calibration. Validating on the robotic instrument segmentation dataset of MICCAI endoscopic vision challenge, our model significantly outperforms state-of-the-art segmentation and detection models, including best-performed models in the challenge.


page 1

page 2

page 4

page 5

page 6


ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery

Representation learning of the task-oriented attention while tracking in...

Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

Global and local relational reasoning enable scene understanding models ...

Learning Where to Look While Tracking Instruments in Robot-assisted Surgery

Directing of the task-specific attention while tracking instrument in su...

Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery

Segmentation of surgical instruments is an important problem in robot-as...

A Nested U-Structure for Instrument Segmentation in Robotic Surgery

Robot-assisted surgery has made great progress with the development of m...

Task-Aware Asynchronous Multi-Task Model with Class Incremental Contrastive Learning for Surgical Scene Understanding

Purpose: Surgery scene understanding with tool-tissue interaction recogn...

RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics

Intelligent robots are designed to effectively navigate dynamic and unpr...

Please sign up or login with your details

Forgot password? Click here to reset