Boomerang: Local sampling on image manifolds using diffusion models

by   Lorenzo Luzi, et al.

Diffusion models can be viewed as mapping points in a high-dimensional latent space onto a low-dimensional learned manifold, typically an image manifold. The intermediate values between the latent space and image manifold can be interpreted as noisy images which are determined by the noise scheduling scheme employed during pre-training. We exploit this interpretation to introduce Boomerang, a local image manifold sampling approach using the dynamics of diffusion models. We call it Boomerang because we first add noise to an input image, moving it closer to the latent space, then bring it back to the image space through diffusion dynamics. We use this method to generate images which are similar, but nonidentical, to the original input images on the image manifold. We are able to set how close the generated image is to the original based on how much noise we add. Additionally, the generated images have a degree of stochasticity, allowing us to locally sample as many times as we want without repetition. We show three applications for which Boomerang can be used. First, we provide a framework for constructing privacy-preserving datasets having controllable degrees of anonymity. Second, we show how to use Boomerang for data augmentation while staying on the image manifold. Third, we introduce a framework for image super-resolution with 8x upsampling. Boomerang does not require any modification to the training of diffusion models and can be used with pretrained models on a single, inexpensive GPU.


page 2

page 6

page 9

page 14

page 15


Creating High Resolution Images with a Latent Adversarial Generator

Generating realistic images is difficult, and many formulations for this...

PartDiff: Image Super-resolution with Partial Diffusion Models

Denoising diffusion probabilistic models (DDPMs) have achieved impressiv...

Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models

This work aims to improve the applicability of diffusion models in reali...

Learning Low Bending and Low Distortion Manifold Embeddings: Theory and Applications

Autoencoders, which consist of an encoder and a decoder, are widely used...

Unsupervised Out-of-Distribution Detection with Diffusion Inpainting

Unsupervised out-of-distribution detection (OOD) seeks to identify out-o...

Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later

How do diffusion generative models convert pure noise into meaningful im...

Training on Thin Air: Improve Image Classification with Generated Data

Acquiring high-quality data for training discriminative models is a cruc...

Please sign up or login with your details

Forgot password? Click here to reset