Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

by   Dominik Rivoir, et al.

Research in unpaired video translation has mainly focused on short-term temporal consistency by conditioning on neighboring frames. However for transfer from simulated to photorealistic sequences, available information on the underlying geometry offers potential for achieving global consistency across views. We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translations of arbitrary views and thus enables long-term consistent video synthesis. We design and test our model to generate video sequences from minimally-invasive surgical abdominal scenes. Because labeled data is often limited in this domain, photorealistic data where ground truth information from the simulated domain is preserved is especially relevant. By extending existing image-based methods to view-consistent videos, we aim to impact the applicability of simulated training and evaluation environments for surgical applications. Code and data will be made publicly available soon.


page 1

page 4

page 6

page 7

page 8


World-Consistent Video-to-Video Synthesis

Video-to-video synthesis (vid2vid) aims for converting high-level semant...

Temporally Consistent Video Transformer for Long-Term Video Prediction

Generating long, temporally consistent video remains an open challenge i...

Learning Blind Video Temporal Consistency

Applying image processing algorithms independently to each frame of a vi...

Novel View Synthesis with Diffusion Models

We present 3DiM, a diffusion model for 3D novel view synthesis, which is...

Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

Colorectal cancer screening modalities, such as optical colonoscopy (OC)...

SurgeonAssist-Net: Towards Context-Aware Head-Mounted Display-Based Augmented Reality for Surgical Guidance

We present SurgeonAssist-Net: a lightweight framework making action-and-...

Please sign up or login with your details

Forgot password? Click here to reset