A Formal Characterization of Black-Box System Safety Performance with Scenario Sampling

by   Bowen Weng, et al.

A typical scenario-based evaluation framework seeks to characterize a black-box system's safety performance (e.g., failure rate) through repeatedly sampling initialization configurations (scenario sampling) and executing a certain test policy for scenario propagation (scenario testing) with the black-box system involved as the test subject. In this letter, we first present a novel safety evaluation criterion that seeks to characterize the actual operational domain within which the test subject would remain safe indefinitely with high probability. By formulating the black-box testing scenario as a dynamic system, we show that the presented problem is equivalent to finding a certain "almost" robustly forward invariant set for the given system. Second, for an arbitrary scenario testing strategy, we propose a scenario sampling algorithm that is provably asymptotically optimal in obtaining the safe invariant set with arbitrarily high accuracy. Moreover, as one considers different testing strategies (e.g., biased sampling of safety-critical cases), we show that the proposed algorithm still converges to the unbiased approximation of the safety characterization outcome if the scenario testing satisfies a certain condition. Finally, the effectiveness of the presented scenario sampling algorithms and various theoretical properties are demonstrated in a case study of the safety evaluation of a control barrier function-based mobile robot collision avoidance system.


page 1

page 8


Rethink the Adversarial Scenario-based Safety Testing of Robots: the Comparability and Optimal Aggressiveness

This paper studies the class of scenario-based safety testing algorithms...

Towards Guaranteed Safety Assurance of Automated Driving Systems with Scenario Sampling: An Invariant Set Perspective (Extended Version)

How many scenarios are sufficient to validate the safe Operational Desig...

On Safety Testing, Validation, and Characterization with Scenario-Sampling: A Case Study of Legged Robots

The dynamic response of the legged robot locomotion is non-Lipschitz and...

Safe non-smooth black-box optimization with application to policy search

For safety-critical black-box optimization tasks, observations of the co...

A Theory of Black-Box Tests

The purpose of testing a system with respect to a requirement is to refu...

Log Barriers for Safe Non-convex Black-box Optimization

We address the problem of minimizing a smooth function f^0(x) over a com...

Efficient Determination of Safety Requirements for Perception Systems

Perception systems operate as a subcomponent of the general autonomy sta...

Please sign up or login with your details

Forgot password? Click here to reset