Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models

by   Zalan Fabian, et al.

Inverse problems arise in a multitude of applications, where the goal is to recover a clean signal from noisy and possibly (non)linear observations. The difficulty of a reconstruction problem depends on multiple factors, such as the structure of the ground truth signal, the severity of the degradation, the implicit bias of the reconstruction model and the complex interactions between the above factors. This results in natural sample-by-sample variation in the difficulty of a reconstruction task, which is often overlooked by contemporary techniques. Recently, diffusion-based inverse problem solvers have established new state-of-the-art in various reconstruction tasks. However, they have the drawback of being computationally prohibitive. Our key observation in this paper is that most existing solvers lack the ability to adapt their compute power to the difficulty of the reconstruction task, resulting in long inference times, subpar performance and wasteful resource allocation. We propose a novel method that we call severity encoding, to estimate the degradation severity of noisy, degraded signals in the latent space of an autoencoder. We show that the estimated severity has strong correlation with the true corruption level and can give useful hints at the difficulty of reconstruction problems on a sample-by-sample basis. Furthermore, we propose a reconstruction method based on latent diffusion models that leverages the predicted degradation severities to fine-tune the reverse diffusion sampling trajectory and thus achieve sample-adaptive inference times. We utilize latent diffusion posterior sampling to maintain data consistency with observations. We perform experiments on both linear and nonlinear inverse problems and demonstrate that our technique achieves performance comparable to state-of-the-art diffusion-based techniques, with significant improvements in computational efficiency.


page 2

page 6

page 7

page 8

page 9


Fast Diffusion Sampler for Inverse Problems by Geometric Decomposition

Diffusion models have shown exceptional performance in solving inverse p...

DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency

Diffusion models have established new state of the art in a multitude of...

Diffusion Posterior Sampling for General Noisy Inverse Problems

Diffusion models have been recently studied as powerful generative inver...

Direct Diffusion Bridge using Data Consistency for Inverse Problems

Diffusion model-based inverse problem solvers have shown impressive perf...

INDigo: An INN-Guided Probabilistic Diffusion Algorithm for Inverse Problems

Recently it has been shown that using diffusion models for inverse probl...

Improving Diffusion Models for Inverse Problems using Manifold Constraints

Recently, diffusion models have been used to solve various inverse probl...

End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics

High-energy collisions at the Large Hadron Collider (LHC) provide valuab...

Please sign up or login with your details

Forgot password? Click here to reset