The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning

by   Vindula Jayawardana, et al.

Evaluations of Deep Reinforcement Learning (DRL) methods are an integral part of scientific progress of the field. Beyond designing DRL methods for general intelligence, designing task-specific methods is becoming increasingly prominent for real-world applications. In these settings, the standard evaluation practice involves using a few instances of Markov Decision Processes (MDPs) to represent the task. However, many tasks induce a large family of MDPs owing to variations in the underlying environment, particularly in real-world contexts. For example, in traffic signal control, variations may stem from intersection geometries and traffic flow levels. The select MDP instances may thus inadvertently cause overfitting, lacking the statistical power to draw conclusions about the method's true performance across the family. In this article, we augment DRL evaluations to consider parameterized families of MDPs. We show that in comparison to evaluating DRL methods on select MDP instances, evaluating the MDP family often yields a substantially different relative ranking of methods, casting doubt on what methods should be considered state-of-the-art. We validate this phenomenon in standard control benchmarks and the real-world application of traffic signal control. At the same time, we show that accurately evaluating on an MDP family is nontrivial. Overall, this work identifies new challenges for empirical rigor in reinforcement learning, especially as the outcomes of DRL trickle into downstream decision-making.


Exploring applications of deep reinforcement learning for real-world autonomous driving systems

Deep Reinforcement Learning (DRL) has become increasingly powerful in re...

Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Sampled environment transitions are a critical input to deep reinforceme...

A Survey on Deep Reinforcement Learning for Audio-Based Applications

Deep reinforcement learning (DRL) is poised to revolutionise the field o...

Deep Reinforcement Learning Based Networked Control with Network Delays for Signal Temporal Logic Specifications

We present a novel deep reinforcement learning (DRL)-based design of a n...

Congested Urban Networks Tend to Be Insensitive to Signal Settings: Implications for Learning-Based Control

This paper highlights several properties of large urban networks that ca...

Deep Contextual Bandits for Orchestrating Multi-User MISO Systems with Multiple RISs

The emergent technology of Reconfigurable Intelligent Surfaces (RISs) ha...

Classification with Costly Features as a Sequential Decision-Making Problem

This work focuses on a specific classification problem, where the inform...

Please sign up or login with your details

Forgot password? Click here to reset