Hybrid Sequence to Sequence Model for Video Object Segmentation

by   Fatemeh Azimi, et al.

One-shot Video Object Segmentation (VOS) is the task of pixel-wise tracking an object of interest within a video sequence, where the segmentation mask of the first frame is given at inference time. In recent years, Recurrent Neural Networks (RNNs) have been widely used for VOS tasks, but they often suffer from limitations such as drift and error propagation. In this work, we study an RNN-based architecture and address some of these issues by proposing a hybrid sequence-to-sequence architecture named HS2S, utilizing a hybrid mask propagation strategy that allows incorporating the information obtained from correspondence matching. Our experiments show that augmenting the RNN with correspondence matching is a highly effective solution to reduce the drift problem. The additional information helps the model to predict more accurate masks and makes it robust against error propagation. We evaluate our HS2S model on the DAVIS2017 dataset as well as Youtube-VOS. On the latter, we achieve an improvement of 11.2pp in the overall segmentation accuracy over RNN-based state-of-the-art methods in VOS. We analyze our model's behavior in challenging cases such as occlusion and long sequences and show that our hybrid architecture significantly enhances the segmentation quality in these difficult scenarios.


page 1

page 2

page 4

page 7


Revisiting Sequence-to-Sequence Video Object Segmentation with Multi-Task Loss and Skip-Memory

Video Object Segmentation (VOS) is an active research area of the visual...

RVOS: End-to-End Recurrent Network for Video Object Segmentation

Multiple object video object segmentation is a challenging task, special...

Boosting Video Object Segmentation via Space-time Correspondence Learning

Current top-leading solutions for video object segmentation (VOS) typica...

Learning What to Learn for Video Object Segmentation

Video object segmentation (VOS) is a highly challenging problem, since t...

Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation

The objective of this paper is self-supervised learning of video object ...

Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation

Current prevailing Video Object Segmentation (VOS) methods usually perfo...

Curriculum Learning for Recurrent Video Object Segmentation

Video object segmentation can be understood as a sequence-to-sequence ta...

Please sign up or login with your details

Forgot password? Click here to reset