Does Road Diversity Really Matter in Testing Automated Driving Systems? – A Registered Report

by   Stefan Klikovits, et al.

Background/Context. The use of automated driving systems (ADSs) in the real world requires rigorous testing to ensure safety. To increase trust, ADSs should be tested on a large set of diverse road scenarios. Literature suggests that if a vehicle is driven along a set of geometrically diverse roads-measured using various diversity measures (DMs)-it will react in a wide range of behaviours, thereby increasing the chances of observing failures (if any), or strengthening the confidence in its safety, if no failures are observed. To the best of our knowledge, however, this assumption has never been tested before, nor have road DMs been assessed for their properties. Objective/Aim. Our goal is to perform an exploratory study on 47 currently used and new, potentially promising road DMs. Specifically, our research questions look into the road DMs themselves, to analyse their properties (e.g. monotonicity, computation efficiency), and to test correlation between DMs. Furthermore, we look at the use of road DMs to investigate whether the assumption that diverse test suites of roads expose diverse driving behaviour holds. Method. Our empirical analysis relies on a state-of-the-art, open-source ADSs testing infrastructure and uses a data set containing over 97,000 individual road geometries and matching simulation data that were collected using two driving agents. By sampling random test suites of various sizes and measuring their roads' geometric diversity, we study road DMs properties, the correlation between road DMs, and the correlation between road DMs and the observed behaviour.


page 1

page 2

page 3

page 4


Extraction of Road Users' Behavior From Realistic Data According to Assumptions in Safety-Related Models for Automated Driving Systems

In this work, we utilized the methodology outlined in the IEEE Standard ...

Road Network Variation Based on HD Map Analysis for the Simulative Safety Assurance of Automated Vehicles

The validation and verification of automated driving functions (ADFs) is...

Collision Avoidance Testing of the Waymo Automated Driving System

This paper describes Waymo's Collision Avoidance Testing (CAT) methodolo...

A Diversity Analysis of Safety Metrics Comparing Vehicle Performance in the Lead-Vehicle Interaction Regime

Vehicle performance metrics analyze data sets consisting of subject vehi...

Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation

Simulation is a crucial tool for accelerating the development of autonom...

Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes

In the life cycle of highly automated systems operating in an open and d...

DriCon: On-device Just-in-Time Context Characterization for Unexpected Driving Events

Driving is a complex task carried out under the influence of diverse spa...

Please sign up or login with your details

Forgot password? Click here to reset