AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies

by   Weiyan Shi, et al.

Existing approaches built separate classifiers to detect nonsense in dialogues. In this paper, we show that without external classifiers, dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond "I don't understand" to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models. Results also show that one single reply without much computation overheads can also detect dialogue nonsense reasonably well.


page 1

page 2

page 3

page 4


DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning

Dialogue State Tracking (DST), a key component of task-oriented conversa...

Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization

A series of datasets and models have been proposed for summaries generat...

"Wait, I'm Still Talking!" Predicting the Dialogue Interaction Behavior Using Imagine-Then-Arbitrate Model

Producing natural and accurate responses like human beings is the ultima...

A Benchmark for Understanding Dialogue Safety in Mental Health Support

Dialogue safety remains a pervasive challenge in open-domain human-machi...

Learning how to learn: an adaptive dialogue agent for incrementally learning visually grounded word meanings

We present an optimised multi-modal dialogue agent for interactive learn...

Using Ontology-Based Context in the Portuguese-English Translation of Homographs in Textual Dialogues

This paper introduces a novel approach to tackle the existing gap on mes...

Please sign up or login with your details

Forgot password? Click here to reset