Video Frame Interpolation with Stereo Event and Intensity Camera

by   Chao Ding, et al.

The stereo event-intensity camera setup is widely applied to leverage the advantages of both event cameras with low latency and intensity cameras that capture accurate brightness and texture information. However, such a setup commonly encounters cross-modality parallax that is difficult to be eliminated solely with stereo rectification especially for real-world scenes with complex motions and varying depths, posing artifacts and distortion for existing Event-based Video Frame Interpolation (E-VFI) approaches. To tackle this problem, we propose a novel Stereo Event-based VFI (SE-VFI) network (SEVFI-Net) to generate high-quality intermediate frames and corresponding disparities from misaligned inputs consisting of two consecutive keyframes and event streams emitted between them. Specifically, we propose a Feature Aggregation Module (FAM) to alleviate the parallax and achieve spatial alignment in the feature domain. We then exploit the fused features accomplishing accurate optical flow and disparity estimation, and achieving better interpolated results through flow-based and synthesis-based ways. We also build a stereo visual acquisition system composed of an event camera and an RGB-D camera to collect a new Stereo Event-Intensity Dataset (SEID) containing diverse scenes with complex motions and varying depths. Experiments on public real-world stereo datasets, i.e., DSEC and MVSEC, and our SEID dataset demonstrate that our proposed SEVFI-Net outperforms state-of-the-art methods by a large margin.


page 1

page 7

page 8

page 10

page 12

page 13


Learning Parallax for Stereo Event-based Motion Deblurring

Due to the extremely low latency, events have been recently exploited to...

Event Fusion Photometric Stereo Network

We introduce a novel method to estimate surface normal of an object in a...

Revisiting Event-based Video Frame Interpolation

Dynamic vision sensors or event cameras provide rich complementary infor...

TimeLens: Event-based Video Frame Interpolation

State-of-the-art frame interpolation methods generate intermediate frame...

Learning Optical Flow from Event Camera with Rendered Dataset

We study the problem of estimating optical flow from event cameras. One ...

Self-Supervised Intensity-Event Stereo Matching

Event cameras are novel bio-inspired vision sensors that output pixel-le...

An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras

Event cameras are ideally suited to capture High Dynamic Range (HDR) vis...

Please sign up or login with your details

Forgot password? Click here to reset