Bayesian Image Reconstruction using Deep Generative Models

by   Razvan V. Marinescu, et al.

Machine learning models are commonly trained end-to-end and in a supervised setting, using paired (input, output) data. Classical examples include recent super-resolution methods that train on pairs of (low-resolution, high-resolution) images. However, these end-to-end approaches require re-training every time there is a distribution shift in the inputs (e.g., night images vs daylight) or relevant latent variables (e.g., camera blur or hand motion). In this work, we leverage state-of-the-art (SOTA) generative models (here StyleGAN2) for building powerful image priors, which enable application of Bayes' theorem for many downstream reconstruction tasks. Our method, called Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks, i.e., super-resolution and in-painting, by combining it with different forward corruption models. We demonstrate BRGM on three large, yet diverse, datasets that enable us to build powerful priors: (i) 60,000 images from the Flick Faces High Quality dataset (ii) 240,000 chest X-rays from MIMIC III and (iii) a combined collection of 5 brain MRI datasets with 7,329 scans. Across all three datasets and without any dataset-specific hyperparameter tuning, our approach yields state-of-the-art performance on super-resolution, particularly at low-resolution levels, as well as inpainting, compared to state-of-the-art methods that are specific to each reconstruction task. Our source code and all pre-trained models are available online:


page 5

page 8

page 12

page 13

page 14

page 15

page 17

page 18


InverseSR: 3D Brain MRI Super-Resolution Using a Latent Diffusion Model

High-resolution (HR) MRI scans obtained from research-grade medical cent...

Generative Models for Generic Light Field Reconstruction

Recently deep generative models have achieved impressive progress in mod...

Deep learning architectural designs for super-resolution of noisy images

Recent advances in deep learning have led to significant improvements in...

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond

We show that pre-trained Generative Adversarial Networks (GANs) such as ...

MrSARP: A Hierarchical Deep Generative Prior for SAR Image Super-resolution

Generative models learned from training using deep learning methods can ...

Self-Validation: Early Stopping for Single-Instance Deep Generative Priors

Recent works have shown the surprising effectiveness of deep generative ...

Regularization via deep generative models: an analysis point of view

This paper proposes a new way of regularizing an inverse problem in imag...

Please sign up or login with your details

Forgot password? Click here to reset