Dimensionality-Varying Diffusion Process

by   Han Zhang, et al.

Diffusion models, which learn to reverse a signal destruction process to generate new data, typically require the signal at each step to have the same dimension. We argue that, considering the spatial redundancy in image signals, there is no need to maintain a high dimensionality in the evolution process, especially in the early generation phase. To this end, we make a theoretical generalization of the forward diffusion process via signal decomposition. Concretely, we manage to decompose an image into multiple orthogonal components and control the attenuation of each component when perturbing the image. That way, along with the noise strength increasing, we are able to diminish those inconsequential components and thus use a lower-dimensional signal to represent the source, barely losing information. Such a reformulation allows to vary dimensions in both training and inference of diffusion models. Extensive experiments on a range of datasets suggest that our approach substantially reduces the computational cost and achieves on-par or even better synthesis performance compared to baseline methods. We also show that our strategy facilitates high-resolution image synthesis and improves FID of diffusion model trained on FFHQ at 1024×1024 resolution from 52.40 to 10.46. Code and models will be made publicly available.


page 1

page 2

page 9


Relay Diffusion: Unifying diffusion process across resolutions for image synthesis

Diffusion models achieved great success in image synthesis, but still fa...

Cartoondiff: Training-free Cartoon Image Generation with Diffusion Transformer Models

Image cartoonization has attracted significant interest in the field of ...

Trans-Dimensional Generative Modeling via Jump Diffusion Models

We propose a new class of generative models that naturally handle data o...

Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later

How do diffusion generative models convert pure noise into meaningful im...

Decoupled Diffusion Models with Explicit Transition Probability

Recent diffusion probabilistic models (DPMs) have shown remarkable abili...

On the Design Fundamentals of Diffusion Models: A Survey

Diffusion models are generative models, which gradually add and remove n...

On the Importance of Noise Scheduling for Diffusion Models

We empirically study the effect of noise scheduling strategies for denoi...

Please sign up or login with your details

Forgot password? Click here to reset