DeepAI AI Chat
Log In Sign Up

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

by   Lorenz Kuhn, et al.
University of Oxford

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" – different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy – an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.


page 1

page 2

page 3

page 4


Cognitive Systems and Question Answering

This paper briefly characterizes the field of cognitive computing. As an...

Uncertainty in Natural Language Generation: From Theory to Applications

Recent advances of powerful Language Models have allowed Natural Languag...

Incidental Supervision from Question-Answering Signals

Human annotations are costly for many natural language processing (NLP) ...

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

In the realm of natural language processing, the understanding of tabula...

Ensemble approach for natural language question answering problem

Machine comprehension, answering a question depending on a given context...

CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Models

Text classifiers built on Pre-trained Language Models (PLMs) have achiev...

Semantic Consistency for Assuring Reliability of Large Language Models

Large Language Models (LLMs) exhibit remarkable fluency and competence a...