High Fidelity Interactive Video Segmentation Using Tensor Decomposition Boundary Loss Convolutional Tessellations and Context Aware Skip Connections

by   Anthony D. Rhodes, et al.

We provide a high fidelity deep learning algorithm (HyperSeg) for interactive video segmentation tasks using a convolutional network with context-aware skip connections, and compressed, hypercolumn image features combined with a convolutional tessellation procedure. In order to maintain high output fidelity, our model crucially processes and renders all image features in high resolution, without utilizing downsampling or pooling procedures. We maintain this consistent, high grade fidelity efficiently in our model chiefly through two means: (1) We use a statistically-principled tensor decomposition procedure to modulate the number of hypercolumn features and (2) We render these features in their native resolution using a convolutional tessellation technique. For improved pixel level segmentation results, we introduce a boundary loss function; for improved temporal coherence in video data, we include temporal image information in our model. Through experiments, we demonstrate the improved accuracy of our model against baseline models for interactive segmentation tasks using high resolution video data. We also introduce a benchmark video segmentation dataset, the VFX Segmentation Dataset, which contains over 27,046 high resolution video frames, including greenscreen and various composited scenes with corresponding, hand crafted, pixel level segmentations. Our work presents an extension to improvement to state of the art segmentation fidelity with high resolution data and can be used across a broad range of application domains, including VFX pipelines and medical imaging disciplines.


page 1

page 5

page 6

page 7


HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator

Video prediction is an important yet challenging problem; burdened with ...

HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

Representing human performance at high-fidelity is an essential building...

Time-Space Transformers for Video Panoptic Segmentation

We propose a novel solution for the task of video panoptic segmentation,...

Temporal-controlled Frame Swap for Generating High-Fidelity Stereo Driving Data for Autonomy Analysis

This paper presents a novel approach, TeFS (Temporal-controlled Frame Sw...

Learning High-fidelity Light Field Images From Hybrid Inputs

This paper explores the reconstruction of high-fidelity LF images (i.e.,...

Digital Elevation Model enhancement using Deep Learning

We demonstrate high fidelity enhancement of planetary digital elevation ...

Learning to Refine Object Segments

Object segmentation requires both object-level information and low-level...

Please sign up or login with your details

Forgot password? Click here to reset