Scenario-based Evaluation of Prediction Models for Automated Vehicles

by   Manuel Muñoz Sánchez, et al.

To operate safely, an automated vehicle (AV) must anticipate how the environment around it will evolve. For that purpose, it is important to know which prediction models are most appropriate for every situation. Currently, assessment of prediction models is often performed over a set of trajectories without distinction of the type of movement they capture, resulting in the inability to determine the suitability of each model for different situations. In this work we illustrate how standardized evaluation methods result in wrong conclusions regarding a model's predictive capabilities, preventing a clear assessment of prediction models and potentially leading to dangerous on-road situations. We argue that following evaluation practices in safety assessment for AVs, assessment of prediction models should be performed in a scenario-based fashion. To encourage scenario-based assessment of prediction models and illustrate the dangers of improper assessment, we categorize trajectories of the Waymo Open Motion dataset according to the type of movement they capture. Next, three different models are thoroughly evaluated for different trajectory types and prediction horizons. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.


MEAT: Maneuver Extraction from Agent Trajectories

Advances in learning-based trajectory prediction are enabled by large-sc...

Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios

Autonomous vehicles currently suffer from a time-inefficient driving sty...

Exploring to establish an appropriate model for mage aesthetic assessment via CNN-based RSRL: An empirical study

To establish an appropriate model for photo aesthetic assessment, in thi...

SafeShift: Safety-Informed Distribution Shifts for Robust Trajectory Prediction in Autonomous Driving

As autonomous driving technology matures, safety and robustness of its k...

A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation

Behavior prediction remains one of the most challenging tasks in the aut...

Post-Selection Confidence Bounds for Prediction Performance

In machine learning, the selection of a promising model from a potential...

A comprehensive study on the prediction reliability of graph neural networks for virtual screening

Prediction models based on deep neural networks are increasingly gaining...

Please sign up or login with your details

Forgot password? Click here to reset