AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

by   Tong Wu, et al.

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be 100×∼600× faster when achieving comparable results. Our code is available at


page 1

page 2

page 3

page 4


Self-conditioned Embedding Diffusion for Text Generation

Can continuous diffusion models bring the same performance breakthrough ...

LLMCad: Fast and Scalable On-device Large Language Model Inference

Generative tasks, such as text generation and question answering, hold a...

On Efficient Training, Controllability and Compositional Generalization of Insertion-based Language Generators

Auto-regressive language models with the left-to-right generation order ...

Step-unrolled Denoising Autoencoders for Text Generation

In this paper we propose a new generative model of text, Step-unrolled D...

Progressive Denoising Model for Fine-Grained Text-to-Image Generation

Recently, vector quantized autoregressive (VQ-AR) models have shown rema...

DiffusER: Discrete Diffusion via Edit-based Reconstruction

In text generation, models that generate text from scratch one token at ...

Towards More Efficient Insertion Transformer with Fractional Positional Encoding

Auto-regressive neural sequence models have been shown to be effective a...

Please sign up or login with your details

Forgot password? Click here to reset