4D-OR: Semantic Scene Graphs for OR Domain Modeling

by   Ege Özsoy, et al.

Surgical procedures are conducted in highly complex operating rooms (OR), comprising different actors, devices, and interactions. To date, only medically trained human experts are capable of understanding all the links and interactions in such a demanding environment. This paper aims to bring the community one step closer to automated, holistic and semantic understanding and modeling of OR domain. Towards this goal, for the first time, we propose using semantic scene graphs (SSG) to describe and summarize the surgical scene. The nodes of the scene graphs represent different actors and objects in the room, such as medical staff, patients, and medical equipment, whereas edges are the relationships between them. To validate the possibilities of the proposed representation, we create the first publicly available 4D surgical SSG dataset, 4D-OR, containing ten simulated total knee replacement surgeries recorded with six RGB-D sensors in a realistic OR simulation center. 4D-OR includes 6734 frames and is richly annotated with SSGs, human and object poses, and clinical roles. We propose an end-to-end neural network-based SSG generation pipeline, with a rate of success of 0.75 macro F1, indeed being able to infer semantic reasoning in the OR. We further demonstrate the representation power of our scene graphs by using it for the problem of clinical role prediction, where we achieve 0.85 macro F1. The code and dataset will be made available upon acceptance.


page 4

page 5


LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Modern surgeries are performed in complex and dynamic settings, includin...

m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks

Autonomous surgical procedures, in particular minimal invasive surgeries...

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Learning to infer graph representations and performing spatial reasoning...

Towards Holistic Surgical Scene Understanding

Most benchmarks for studying surgical interventions focus on a specific ...

Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

Global and local relational reasoning enable scene understanding models ...

Semantic segmentation of surgical hyperspectral images under geometric domain shifts

Robust semantic segmentation of intraoperative image data could pave the...

Secondary Use of Clinical Problem List Entries for Neural Network-Based Disease Code Assignment

Clinical information systems have become large repositories for semi-str...

Please sign up or login with your details

Forgot password? Click here to reset