ssROC: Semi-Supervised ROC Analysis for Reliable and Streamlined Evaluation of Phenotyping Algorithms

by   Jianhui Gao, et al.

Objective: High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed to estimate PAs. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (e.g., sensitivity, specificity). Materials and Methods: ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC through in-depth simulation studies and an extensive evaluation of eight PAs from Mass General Brigham. Results: In both simulated and real data, ssROC produced ROC parameter estimates with significantly lower variance than supROC for a given amount of labeled data. For the eight PAs, our results illustrate that ssROC achieves similar precision to supROC, but with approximately 60 of labeled data on average. Discussion: ssROC enables precise evaluation of PA performance to increase trust in observational health research without demanding large volumes of labeled data. ssROC is also easily implementable in open-source software. Conclusion: When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.


page 12

page 18

page 20


Defect detection using weakly supervised learning

In many real-world scenarios, obtaining large amounts of labeled data ca...

Audio Event and Scene Recognition: A Unified Approach using Strongly and Weakly Labeled Data

In this paper we propose a novel learning framework called Supervised an...

Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance

In many modern machine learning applications, the outcome is expensive o...

Prior Adaptive Semi-supervised Learning with Application to EHR Phenotyping

Electronic Health Records (EHR) data, a rich source for biomedical resea...

Neural Medication Extraction: A Comparison of Recent Models in Supervised and Semi-supervised Learning Settings

Drug prescriptions are essential information that must be encoded in ele...

Query-Adaptive Predictive Inference with Partial Labels

The cost and scarcity of fully supervised labels in statistical machine ...

Conformal Prediction with Partially Labeled Data

While the predictions produced by conformal prediction are set-valued, t...

Please sign up or login with your details

Forgot password? Click here to reset