Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs

by   Giorgio Giannone, et al.

Event-based cameras are novel, efficient sensors inspired by the human vision system, generating an asynchronous, pixel-wise stream of data. Learning from such data is generally performed through heavy preprocessing and event integration into images. This requires buffering of possibly long sequences and can limit the response time of the inference system. In this work, we instead propose to directly use events from a DVS camera, a stream of intensity changes and their spatial coordinates. This sequence is used as the input for a novel asynchronous RNN-like architecture, the Input-filtering Neural ODEs (INODE). This is inspired by the dynamical systems and filtering literature. INODE is an extension of Neural ODEs (NODE) that allows for input signals to be continuously fed to the network, like in filtering. The approach naturally handles batches of time series with irregular time-stamps by implementing a batch forward Euler solver. INODE is trained like a standard RNN, it learns to discriminate short event sequences and to perform event-by-event online inference. We demonstrate our approach on a series of classification tasks, comparing against a set of LSTM baselines. We show that, independently of the camera resolution, INODE can outperform the baselines by a large margin on the ASL task and it's on par with a much larger LSTM for the NCALTECH task. Finally, we show that INODE is accurate even when provided with very few events.


page 1

page 2

page 3

page 4


Asynchronous "Events" are Better For Motion Estimation

Event-based camera is a bio-inspired vision sensor that records intensit...

SE-Harris and eSUSAN: Asynchronous Event-Based Corner Detection Using Megapixel Resolution CeleX-V Camera

Event cameras are novel neuromorphic vision sensors with ultrahigh tempo...

High Speed and High Dynamic Range Video with an Event Camera

Event cameras are novel sensors that report brightness changes in the fo...

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Learning coordinate-based volumetric 3D scene representations such as ne...

Exploiting Spatial Sparsity for Event Cameras with Visual Transformers

Event cameras report local changes of brightness through an asynchronous...

Event-LSTM: An Unsupervised and Asynchronous Learning-based Representation for Event-based Data

Event cameras are activity-driven bio-inspired vision sensors, thereby r...

Real-Time Pose Estimation for Event Cameras with Stacked Spatial LSTM Networks

We present a new method to estimate the 6DOF pose of the event camera so...

Please sign up or login with your details

Forgot password? Click here to reset