The Misuse of AUC: What High Impact Risk Assessment Gets Wrong

by   Kweku Kwegyir-Aggrey, et al.

When determining which machine learning model best performs some high impact risk assessment task, practitioners commonly use the Area under the Curve (AUC) to defend and validate their model choices. In this paper, we argue that the current use and understanding of AUC as a model performance metric misunderstands the way the metric was intended to be used. To this end, we characterize the misuse of AUC and illustrate how this misuse negatively manifests in the real world across several risk assessment domains. We locate this disconnect in the way the original interpretation of AUC has shifted over time to the point where issues pertaining to decision thresholds, class balance, statistical uncertainty, and protected groups remain unaddressed by AUC-based model comparisons, and where model choices that should be the purview of policymakers are hidden behind the veil of mathematical rigor. We conclude that current model validation practices involving AUC are not robust, and often invalid.


page 1

page 2

page 3

page 4


ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (RO...

A Distributionally Robust Area Under Curve Maximization Model

Area under ROC curve (AUC) is a widely used performance measure for clas...

Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation

Optimal performance is critical for decision-making tasks from medicine ...

CatBoost model with synthetic features in application to loan risk assessment of small businesses

Loan risk for small businesses has long been a complex problem worthy of...

Revealing posturographic features associated with the risk of falling in patients with Parkinsonian syndromes via machine learning

Falling in Parkinsonian syndromes (PS) is associated with postural insta...

Limitations of Pinned AUC for Measuring Unintended Bias

This report examines the Pinned AUC metric introduced and highlights som...

Please sign up or login with your details

Forgot password? Click here to reset