Differentiable Learning Under Triage

03/16/2021
by   Nastaran Okati, et al.
0

Multiple lines of evidence suggest that predictive models may benefit from algorithmic triage. Under algorithmic triage, a predictive model does not predict all instances but instead defers some of them to human experts. However, the interplay between the prediction accuracy of the model and the human experts under algorithmic triage is not well understood. In this work, we start by formally characterizing under which circumstances a predictive model may benefit from algorithmic triage. In doing so, we also demonstrate that models trained for full automation may be suboptimal under triage. Then, given any model and desired level of triage, we show that the optimal triage policy is a deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on a per-instance level. Building upon these results, we introduce a practical gradient-based algorithm that is guaranteed to find a sequence of triage policies and predictive models of increasing performance. Experiments on a wide variety of supervised learning tasks using synthetic and real data from two important applications – content moderation and scientific discovery – illustrate our theoretical results and show that the models and triage policies provided by our gradient-based algorithm outperform those provided by several competitive baselines.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/08/2019

Improving Consequential Decision Making under Imperfect Predictions

Consequential decisions are increasingly informed by sophisticated data-...
research
09/23/2021

Reinforcement Learning Under Algorithmic Triage

Methods to learn under algorithmic triage have predominantly focused on ...
research
06/21/2020

Classification Under Human Assistance

Most supervised learning models are trained for full automation. However...
research
03/28/2019

The Algorithmic Automation Problem: Prediction, Triage, and Human Effort

In a wide array of areas, algorithms are matching and surpassing the per...
research
05/20/2022

How Useful are Gradients for OOD Detection Really?

One critical challenge in deploying highly performant machine learning m...
research
06/02/2023

Auditing for Human Expertise

High-stakes prediction tasks (e.g., patient diagnosis) are often handled...

Please sign up or login with your details

Forgot password? Click here to reset