Self-Supervised Intensity-Event Stereo Matching

by   Jinjin Gu, et al.

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes in microsecond accuracy with a high dynamic range and low power consumption. Despite these advantages, event cameras cannot be directly applied to computational imaging tasks due to the inability to obtain high-quality intensity and events simultaneously. This paper aims to connect a standalone event camera and a modern intensity camera so that the applications can take advantage of both two sensors. We establish this connection through a multi-modal stereo matching task. We first convert events to a reconstructed image and extend the existing stereo networks to this multi-modality condition. We propose a self-supervised method to train the multi-modal stereo network without using ground truth disparity data. The structure loss calculated on image gradients is used to enable self-supervised learning on such multi-modal data. Exploiting the internal stereo constraint between views with different modalities, we introduce general stereo loss functions, including disparity cross-consistency loss and internal disparity loss, leading to improved performance and robustness compared to existing approaches. The experiments demonstrate the effectiveness of the proposed method, especially the proposed general stereo loss functions, on both synthetic and real datasets. At last, we shed light on employing the aligned events and intensity images in downstream tasks, e.g., video interpolation application.


page 5

page 7

page 8

page 13

page 18

page 19

page 20

page 21


Learning Parallax for Stereo Event-based Motion Deblurring

Due to the extremely low latency, events have been recently exploited to...

Stereo Event Lifetime and Disparity Estimation for Dynamic Vision Sensors

Event-based cameras are biologically inspired sensors that output asynch...

Stereo Hybrid Event-Frame (SHEF) Cameras for 3D Perception

Stereo camera systems play an important role in robotics applications to...

Video Frame Interpolation with Stereo Event and Intensity Camera

The stereo event-intensity camera setup is widely applied to leverage th...

Rethinking Cross-Entropy Loss for Stereo Matching Networks

Despite the great success of deep learning in stereo matching, recoverin...

Degradation-agnostic Correspondence from Resolution-asymmetric Stereo

In this paper, we study the problem of stereo matching from a pair of im...

Disparity-based HDR imaging

High-dynamic range imaging permits to extend the dynamic range of intens...

Please sign up or login with your details

Forgot password? Click here to reset