DeepAI AI Chat
Log In Sign Up

CLAM: Selective Clarification for Ambiguous Questions with Large Language Models

by   Lorenz Kuhn, et al.
University of Oxford

State-of-the-art language models are often accurate on many question-answering benchmarks with well-defined questions. Yet, in real settings questions are often unanswerable without asking the user for clarifying information. We show that current SotA models often do not ask the user for clarification when presented with imprecise questions and instead provide incorrect answers or "hallucinate". To address this, we introduce CLAM, a framework that first uses the model to detect ambiguous questions, and if an ambiguous question is detected, prompts the model to ask the user for clarification. Furthermore, we show how to construct a scalable and cost-effective automatic evaluation protocol using an oracle language model with privileged information to provide clarifying information. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set derived from TriviaQA.


page 1

page 2

page 3

page 4


Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions

Many open-domain questions are under-specified and thus have multiple po...

Selectively Answering Ambiguous Questions

Trustworthy language models should abstain from answering questions when...

Talk2Data: High-Level Question Decomposition for Data-Oriented Question and Answering

Through a data-oriented question and answering system, users can directl...

Interactive Classification by Asking Informative Questions

Natural language systems often rely on a single, potentially ambiguous i...

Discovering Latent Knowledge in Language Models Without Supervision

Existing techniques for training language models can be misaligned with ...

Answering Ambiguous Questions via Iterative Prompting

In open-domain question answering, due to the ambiguity of questions, mu...

Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction

In open-domain question answering, questions are highly likely to be amb...