Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interface

by   Michael Desmond, et al.

Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels has not yet been explored. We designed an AI labeling assistant that uses a semi-supervised learning algorithm to predict the most probable labels for each example. We leverage these predictions to provide assistance in two ways: (i) providing a label recommendation and (ii) reducing the labeler's decision space by focusing their attention on only the most probable labels. We conducted a user study (n=54) to evaluate an AI-assisted interface for data labeling in this context. Our results highlight that the AI assistance improves both labeler accuracy and speed, especially when the labeler finds the correct label in the reduced label space. We discuss findings related to the presentation of AI assistance and design implications for intelligent labeling interfaces.


page 5

page 7


Adaptive interventions for both accuracy and time in AI-assisted human decision making

In settings where users are both time-pressured and need high accuracy, ...

How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

Data analysis is challenging as analysts must navigate nuanced decisions...

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making

In AI-assisted decision-making, a central promise of putting a human in ...

Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

Although AI holds promise for improving human decision making in societa...

MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images

The lack of annotated datasets is a major challenge in training new task...

TagLab: A human-centric AI system for interactive semantic segmentation

Fully automatic semantic segmentation of highly specific semantic classe...

CARE: Collaborative AI-Assisted Reading Environment

Recent years have seen impressive progress in AI-assisted writing, yet t...

Please sign up or login with your details

Forgot password? Click here to reset