A Variational Perspective on Solving Inverse Problems with Diffusion Models

by   Morteza Mardani, et al.

Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-Diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for image restoration tasks such as inpainting and superresolution demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models.


page 2

page 6

page 7

page 8

page 13

page 14

page 15

page 16


Denoising Diffusion Restoration Models

Many interesting tasks in image restoration can be cast as linear invers...

JPEG Artifact Correction using Denoising Diffusion Restoration Models

Diffusion models can be used as learned priors for solving various inver...

Diffusion Posterior Sampling for General Noisy Inverse Problems

Diffusion models have been recently studied as powerful generative inver...

Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Imaging Inverse Problems

Denoising diffusion models have emerged as the go-to framework for solvi...

A Regularized Conditional GAN for Posterior Sampling in Inverse Problems

In inverse problems, one seeks to reconstruct an image from incomplete a...

End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics

High-energy collisions at the Large Hadron Collider (LHC) provide valuab...

Improving Diffusion Models for Inverse Problems using Manifold Constraints

Recently, diffusion models have been used to solve various inverse probl...

Please sign up or login with your details

Forgot password? Click here to reset