Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views

by   Jiyuan Zhang, et al.

The extraction of a clean background image by removing foreground occlusion holds immense practical significance, but it also presents several challenges. Presently, the majority of de-occlusion research focuses on addressing this issue through the extraction and synthesis of discrete images from calibrated camera arrays. Nonetheless, the restoration quality tends to suffer when faced with dense occlusions or high-speed motions due to limited perspectives and motion blur. To successfully remove dense foreground occlusion, an effective multi-view visual information integration approach is required. Introducing the spike camera as a novel type of neuromorphic sensor offers promising capabilities with its ultra-high temporal resolution and high dynamic range. In this paper, we propose an innovative solution for tackling the de-occlusion problem through continuous multi-view imaging using only one spike camera without any prior knowledge of camera intrinsic parameters and camera poses. By rapidly moving the spike camera, we continually capture the dense stream of spikes from the occluded scene. To process the spikes, we build a novel model SpkOccNet, in which we integrate information of spikes from continuous viewpoints within multi-windows, and propose a novel cross-view mutual attention mechanism for effective fusion and refinement. In addition, we contribute the first real-world spike-based dataset S-OCC for occlusion removal. The experimental results demonstrate that our proposed model efficiently removes dense occlusions in diverse scenes while exhibiting strong generalization.


page 1

page 3

page 5


Spike Stream Denoising via Spike Camera Simulation

As a neuromorphic sensor with high temporal resolution, the spike camera...

I See-Through You: A Framework for Removing Foreground Occlusion in Both Sparse and Dense Light Field Images

Light field (LF) camera captures rich information from a scene. Using th...

SpikeCodec: An End-to-end Learned Compression Framework for Spiking Camera

Recently, the bio-inspired spike camera with continuous motion recording...

SpikeCV: Open a Continuous Computer Vision Era

SpikeCV is a new open-source computer vision platform for the spike came...

Seeing Behind Dynamic Occlusions with Event Cameras

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, ...

Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework

Rain removal is important for improving the robustness of outdoor vision...

Event-based Synthetic Aperture Imaging with a Hybrid Network

Synthetic aperture imaging (SAI) is able to achieve the see through effe...

Please sign up or login with your details

Forgot password? Click here to reset