Bayesian Image Reconstruction using Deep Generative Models

12/08/2020
by   Razvan V. Marinescu, et al.
20

Machine learning models are commonly trained end-to-end and in a supervised setting, using paired (input, output) data. Classical examples include recent super-resolution methods that train on pairs of (low-resolution, high-resolution) images. However, these end-to-end approaches require re-training every time there is a distribution shift in the inputs (e.g., night images vs daylight) or relevant latent variables (e.g., camera blur or hand motion). In this work, we leverage state-of-the-art (SOTA) generative models (here StyleGAN2) for building powerful image priors, which enable application of Bayes' theorem for many downstream reconstruction tasks. Our method, called Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks, i.e., super-resolution and in-painting, by combining it with different forward corruption models. We demonstrate BRGM on three large, yet diverse, datasets that enable us to build powerful priors: (i) 60,000 images from the Flick Faces High Quality dataset (ii) 240,000 chest X-rays from MIMIC III and (iii) a combined collection of 5 brain MRI datasets with 7,329 scans. Across all three datasets and without any dataset-specific hyperparameter tuning, our approach yields state-of-the-art performance on super-resolution, particularly at low-resolution levels, as well as inpainting, compared to state-of-the-art methods that are specific to each reconstruction task. Our source code and all pre-trained models are available online: https://razvanmarinescu.github.io/brgm/.

READ FULL TEXT

page 5

page 8

page 12

page 13

page 14

page 15

page 17

page 18

research
08/23/2023

InverseSR: 3D Brain MRI Super-Resolution Using a Latent Diffusion Model

High-resolution (HR) MRI scans obtained from research-grade medical cent...
research
05/13/2020

Generative Models for Generic Light Field Reconstruction

Recently deep generative models have achieved impressive progress in mod...
research
02/09/2021

Deep learning architectural designs for super-resolution of noisy images

Recent advances in deep learning have led to significant improvements in...
research
07/29/2022

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond

We show that pre-trained Generative Adversarial Networks (GANs) such as ...
research
11/30/2022

MrSARP: A Hierarchical Deep Generative Prior for SAR Image Super-resolution

Generative models learned from training using deep learning methods can ...
research
10/23/2021

Self-Validation: Early Stopping for Single-Instance Deep Generative Priors

Recent works have shown the surprising effectiveness of deep generative ...
research
01/21/2021

Regularization via deep generative models: an analysis point of view

This paper proposes a new way of regularizing an inverse problem in imag...

Please sign up or login with your details

Forgot password? Click here to reset