Learning how to approve updates to machine learning algorithms in non-stationary settings

by   Jean Feng, et al.

Machine learning algorithms in healthcare have the potential to continually learn from real-world data generated during healthcare delivery and adapt to dataset shifts. As such, the FDA is looking to design policies that can autonomously approve modifications to machine learning algorithms while maintaining or improving the safety and effectiveness of the deployed models. However, selecting a fixed approval strategy, a priori, can be difficult because its performance depends on the stationarity of the data and the quality of the proposed modifications. To this end, we investigate a learning-to-approve approach (L2A) that uses accumulating monitoring data to learn how to approve modifications. L2A defines a family of strategies that vary in their "optimism”—where more optimistic policies have faster approval rates—and searches over this family using an exponentially weighted average forecaster. To control the cumulative risk of the deployed model, we give L2A the option to abstain from making a prediction and incur some fixed abstention cost instead. We derive bounds on the average risk of the model deployed by L2A, assuming the distributional shifts are smooth. In simulation studies and empirical analyses, L2A tailors the level of optimism for each problem-setting: It learns to abstain when performance drops are common and approve beneficial modifications quickly when the distribution is stable.


page 1

page 2

page 3

page 4


Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep

Successful deployment of machine learning algorithms in healthcare requi...

Conformal Inference for Invariant Risk Minimization

The application of machine learning models can be significantly impeded ...

Risk-Averse Offline Reinforcement Learning

Training Reinforcement Learning (RL) agents in high-stakes applications ...

Tracking the risk of a deployed model and detecting harmful distribution shifts

When deployed in the real world, machine learning models inevitably enco...

Convex Optimization-based Policy Adaptation to Compensate for Distributional Shifts

Many real-world systems often involve physical components or operating e...

Sequential algorithmic modification with test data reuse

After initial release of a machine learning algorithm, the model can be ...

Optimal sports betting strategies in practice: an experimental review

We investigate the most popular approaches to the problem of sports bett...

Please sign up or login with your details

Forgot password? Click here to reset