Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction

by   Peichao Li, et al.

Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images acquired with snapshot mosaic cameras, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets captured by high-resolution, but slow, hyperspectral imaging devices. Image reconstruction is achieved using convolutional neural networks for hyperspectral image super-resolution, followed by cross-talk and leakage correction using a sensor-specific calibration matrix. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality compared to a baseline demosaicking method using linear interpolation. Moreover, the fast processing time of 45 ms of our algorithm to obtain super-resolved RGB or oxygenation saturation maps per image frame for a state-of-the-art snapshot mosaic camera demonstrates the potential for its seamless integration into real-time surgical hyperspectral imaging applications.


page 3

page 9

page 11


Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery

Recently, single gray/RGB image super-resolution reconstruction task has...

Hyperspectral Calibration of Art: Acquisition and Calibration Workflows

Hyperspectral imaging has become an increasingly used tool in the analys...

Evaluation of the potential of Near Infrared Hyperspectral Imaging for monitoring the invasive brown marmorated stink bug

The brown marmorated stink bug (BMSB), Halyomorpha halys, is an invasive...

Spectral-Spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification

Early detection of cancerous tissue is crucial for long-term patient sur...

Ill-posed Surface Emissivity Retrieval from Multi-Geometry Hyperspectral Images using a Hybrid Deep Neural Network

Atmospheric correction is a fundamental task in remote sensing because o...

Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging

Intra-operative measurements of tissue shape and multi/ hyperspectral in...

Probe-based Rapid Hybrid Hyperspectral and Tissue Surface Imaging Aided by Fully Convolutional Networks

Tissue surface shape and reflectance spectra provide rich intra-operativ...

Please sign up or login with your details

Forgot password? Click here to reset