Leveraging Expert Consistency to Improve Algorithmic Decision Support

by   Maria De-Arteaga, et al.

Due to their promise of superior predictive power relative to human assessment, machine learning models are increasingly being used to support high-stakes decisions. However, the nature of the labels available for training these models often hampers the usefulness of predictive models for decision support. In this paper, we explore the use of historical expert decisions as a rich–yet imperfect–source of information, and we show that it can be leveraged to mitigate some of the limitations of learning from observed labels alone. We consider the problem of estimating expert consistency indirectly when each case in the data is assessed by a single expert, and propose influence functions based methodology as a solution to this problem. We then incorporate the estimated expert consistency into the predictive model meant for decision support through an approach we term label amalgamation. This allows the machine learning models to learn from experts in instances where there is expert consistency, and learn from the observed labels elsewhere. We show how the proposed approach can help mitigate common challenges of learning from observed labels alone, reducing the gap between the construct that the algorithm optimizes for and the construct of interest to experts. After providing intuition and theoretical results, we present empirical results in the context of child maltreatment hotline screenings. Here, we find that (1) there are high-risk cases whose risk is considered by the experts but not wholly captured in the target labels used to train a deployed model, and (2) the proposed approach improves recall for these cases.


page 1

page 2

page 3

page 4


Learning under selective labels in the presence of expert consistency

We explore the problem of learning under selective labels in the context...

Designing Closed Human-in-the-loop Deferral Pipelines

In hybrid human-machine deferral frameworks, a classifier can defer unce...

Provably Improving Expert Predictions with Conformal Prediction

Automated decision support systems promise to help human experts solve t...

Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts

Machine learning (ML) models are increasingly being used in application ...

A Machine Learning Framework Towards Transparency in Experts' Decision Quality

Expert workers make non-trivial decisions with significant implications....

Expert decision support system for aeroacoustic classification

This paper presents an expert decision support system for time-invariant...

Expert Kaplan–Meier estimation

The setting of a right-censored random sample subject to contamination i...

Please sign up or login with your details

Forgot password? Click here to reset