Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders

by   Maksim Makarenko, et al.

Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision. State-of-the-art implementations of snapshot hyperspectral imaging rely on bulky, non-integrated, and expensive optical elements, including lenses, spectrometers, and filters. These macroscopic components do not allow fast data processing for, e.g real-time and high-resolution videos. This work introduces Hyplex, a new integrated architecture addressing the limitations discussed above. Hyplex is a CMOS-compatible, fast hyperspectral camera that replaces bulk optics with nanoscale metasurfaces inversely designed through artificial intelligence. Hyplex does not require spectrometers but makes use of conventional monochrome cameras, opening up the possibility for real-time and high-resolution hyperspectral imaging at inexpensive costs. Hyplex exploits a model-driven optimization, which connects the physical metasurfaces layer with modern visual computing approaches based on end-to-end training. We design and implement a prototype version of Hyplex and compare its performance against the state-of-the-art for typical imaging tasks such as spectral reconstruction and semantic segmentation. In all benchmarks, Hyplex reports the smallest reconstruction error. We additionally present what is, to the best of our knowledge, the largest publicly available labeled hyperspectral dataset for semantic segmentation.


page 1

page 2

page 3

page 4

page 5

page 7

page 10

page 11


Hyperspectral Image Segmentation: A Preliminary Study on the Oral and Dental Spectral Image Database (ODSI-DB)

Visual discrimination of clinical tissue types remains challenging, with...

Real-Time Semantic Segmentation using Hyperspectral Images for Mapping Unstructured and Unknown Environments

Autonomous navigation in unstructured off-road environments is greatly i...

A laboratory-created dataset with ground-truth for hyperspectral unmixing evaluation

Spectral unmixing is an important and challenging problem in hyperspectr...

End-to-End Hyperspectral-Depth Imaging with Learned Diffractive Optics

To extend the capabilities of spectral imaging, hyperspectral and depth ...

A New Backbone for Hyperspectral Image Reconstruction

The study of 3D hyperspectral image (HSI) reconstruction refers to the i...

Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection

Depth reconstruction and hyperspectral reflectance reconstruction are tw...

MXR-U-Nets for Real Time Hyperspectral Reconstruction

In recent times, CNNs have made significant contributions to application...

Please sign up or login with your details

Forgot password? Click here to reset