How good does a Defect Predictor need to be to guide Search-Based Software Testing?

by   Anjana Perera, et al.

Defect predictors, static bug detectors and humans inspecting the code can locate the parts of the program that are buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs. However, often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this paper, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the probability of correctly identifying buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. On the other hand, the effect of precision, a measure for false alarms, is not of meaningful practical significance as indicated by a very small effect size. In particular, the SBST technique finds 7.5 less bugs on average (out of 420 bugs) for every 5 In the context of combining defect prediction and SBST, our recommendation for practice is to increase the recall of defect predictors at the expense of precision, while maintaining a precision of at least 75 imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.


page 1

page 2

page 3

page 4


Using Defect Prediction to Improve the Bug Detection Capability of Search-Based Software Testing

Automated test generators, such as search based software testing (SBST) ...

Defect Prediction Guided Search-Based Software Testing

Today, most automated test generators, such as search-based software tes...

Practical Testing of a C99 Compiler Using Output Comparison

A simple technique is presented for testing a C99 compiler, by compariso...

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

Graphically-rich applications such as games are ubiquitous with attracti...

SAVIOR: Towards Bug-Driven Hybrid Testing

Hybrid testing combines fuzz testing and concolic execution. It leverage...

Duplicate-sensitivity Guided Transformation Synthesis for DBMS Correctness Bug Detection

Database Management System (DBMS) plays a core role in modern software f...

Guidelines on Minimum Standards for Developer Verification of Software

Executive Order (EO) 14028, "Improving the Nation's Cybersecurity", 12 M...

Please sign up or login with your details

Forgot password? Click here to reset