"Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?

by   R. Stuart Geiger, et al.

Supervised machine learning, in which models are automatically derived from labeled training data, is only as good as the quality of that data. This study builds on prior work that investigated to what extent 'best practices' around labeling training data were followed in applied ML publications within a single domain (social media platforms). In this paper, we expand by studying publications that apply supervised ML in a far broader spectrum of disciplines, focusing on human-labeled data. We report to what extent a random sample of ML application papers across disciplines give specific details about whether best practices were followed, while acknowledging that a greater range of application fields necessarily produces greater diversity of labeling and annotation methods. Because much of machine learning research and education only focuses on what is done once a "ground truth" or "gold standard" of training data is available, it is especially relevant to discuss issues around the equally-important aspect of whether such data is reliable in the first place. This determination becomes increasingly complex when applied to a variety of specialized fields, as labeling can range from a task requiring little-to-no background knowledge to one that must be performed by someone with career expertise.


page 1

page 5

page 9

page 13

page 18

page 19

page 20

page 21


Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?

Many machine learning projects for new application areas involve teams o...

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Machine Learning (ML) workloads have rapidly grown in importance, but ra...

Rapid Classification of Crisis-Related Data on Social Networks using Convolutional Neural Networks

The role of social media, in particular microblogging platforms such as ...

Survey on the Convergence of Machine Learning and Blockchain

Machine learning (ML) has been pervasively researched nowadays and it ha...

An embedded system for the automated generation of labeled plant images to enable machine learning applications in agriculture

A lack of sufficient training data, both in terms of variety and quantit...

Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research

Visualization for machine learning (VIS4ML) research aims to help expert...

Towards Practices for Human-Centered Machine Learning

"Human-centered machine learning" (HCML) is a term that describes machin...

Please sign up or login with your details

Forgot password? Click here to reset