Evaluating Reinforcement Learning Algorithms in Observational Health Settings

by   Omer Gottesman, et al.

Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.


page 1

page 2

page 3

page 4


Challenges for Reinforcement Learning in Healthcare

Many healthcare decisions involve navigating through a multitude of trea...

Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Sepsis Treatment

The potential of Reinforcement Learning (RL) has been demonstrated throu...

Identifying Distinct, Effective Treatments for Acute Hypotension with SODA-RL: Safely Optimized Diverse Accurate Reinforcement Learning

Hypotension in critical care settings is a life-threatening emergency th...

A Reinforcement Learning Approach to Estimating Long-term Treatment Effects

Randomized experiments (a.k.a. A/B tests) are a powerful tool for estima...

IntelligentPooling: Practical Thompson Sampling for mHealth

In mobile health (mHealth) smart devices deliver behavioral treatments r...

An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare

Reinforcement Learning (RL) has recently been applied to sequential esti...

Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies

Treatment policies learned via reinforcement learning (RL) from observat...

Please sign up or login with your details

Forgot password? Click here to reset