The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

by   Kushal Arora, et al.

State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story generation, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of vocabulary diversity, and self-repetition or copying from the context. In this paper, we postulate that “human-like” generations usually lie in a narrow and nearly flat entropy band, and violation of these entropy bounds correlates with degenerate behavior. Our experiments show that this stable narrow entropy zone exists across models, tasks, and domains and confirm the hypothesis that violations of this zone correlate with degeneration. We then use this insight to propose an entropy-aware decoding algorithm that respects these entropy bounds resulting in less degenerate, more contextual, and "human-like" language generation in open-ended text generation settings.


page 1

page 2

page 3

page 4


On the probability-quality paradox in language generation

When generating natural language from neural probabilistic models, high ...

On Decoding Strategies for Neural Text Generators

When generating text from probabilistic models, the chosen decoding stra...

FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy

Measuring the distance between machine-produced and human language is a ...

Event Transition Planning for Open-ended Text Generation

Open-ended text generation tasks, such as dialogue generation and story ...

DISCO : efficient unsupervised decoding for discrete natural language problems via convex relaxation

In this paper we study test time decoding; an ubiquitous step in almost ...

Momentum Decoding: Open-ended Text Generation As Graph Exploration

Open-ended text generation with autoregressive language models (LMs) is ...

Modeling Emergent Lexicon Formation with a Self-Reinforcing Stochastic Process

We introduce FiLex, a self-reinforcing stochastic process which models f...

Please sign up or login with your details

Forgot password? Click here to reset