An Empirical Study of Clarifying Question-Based Systems

by   Jie Zou, et al.

Search and recommender systems that take the initiative to ask clarifying questions to better understand users' information needs are receiving increasing attention from the research community. However, to the best of our knowledge, there is no empirical study to quantify whether and to what extent users are willing or able to answer these questions. In this work, we conduct an online experiment by deploying an experimental system, which interacts with users by asking clarifying questions against a product repository. We collect both implicit interaction behavior data and explicit feedback from users showing that: (a) users are willing to answer a good number of clarifying questions (11-21 on average), but not many more than that; (b) most users answer questions until they reach the target product, but also a fraction of them stops due to fatigue or due to receiving irrelevant questions; (c) part of the users' answers (12-17 target product; while (d) most of the users (66-84 system helpful towards completing their tasks. Some of the findings of the study contradict current assumptions on simulated evaluations in the field, while they point towards improvements in the evaluation framework and can inspire future interactive search/recommender system designs.


page 1

page 2

page 3

page 4


Recommender May Not Favor Loyal Users

In academic research, recommender systems are often evaluated on benchma...

An empirical study of question discussions on Stack Overflow

Stack Overflow provides a means for developers to exchange knowledge. Wh...

Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?

Though state-of-the-art (SOTA) NLP systems have achieved remarkable perf...

Engaging Audiences in Virtual Museums by Interactively Prompting Guiding Questions

Virtual museums aim to promote access to cultural artifacts. However, th...

Want a Good Answer? Ask a Good Question First!

Community Question Answering (CQA) websites have become valuable reposit...

An Exploratory Analysis of the Impact of Named Ranges on the Debugging Performance of Novice Users

This paper describes an exploratory empirical study of the effect of nam...

An Empirical Study of DNNs Robustification Inefficacy in Protecting Visual Recommenders

Visual-based recommender systems (VRSs) enhance recommendation performan...

Please sign up or login with your details

Forgot password? Click here to reset