CamBench – Cryptographic API Misuse Detection Tool Benchmark Suite

by   Michael Schlichtig, et al.

Context: Cryptographic APIs are often misused in real-world applications. Therefore, many cryptographic API misuse detection tools have been introduced. However, there exists no established reference benchmark for a fair and comprehensive comparison and evaluation of these tools. While there are benchmarks, they often only address a subset of the domain or were only used to evaluate a subset of existing misuse detection tools. Objective: To fairly compare cryptographic API misuse detection tools and to drive future development in this domain, we will devise such a benchmark. Openness and transparency in the generation process are key factors to fairly generate and establish the needed benchmark. Method: We propose an approach where we derive the benchmark generation methodology from the literature which consists of general best practices in benchmarking and domain-specific benchmark generation. A part of this methodology is transparency and openness of the generation process, which is achieved by pre-registering this work. Based on our methodology we design CamBench, a fair "Cryptographic API Misuse Detection Tool Benchmark Suite". We will implement the first version of CamBench limiting the domain to Java, the JCA, and static analyses. Finally, we will use CamBench to compare current misuse detection tools and compare CamBench to related benchmarks of its domain.


page 1

page 2

page 3

page 4


Evaluation of Static Vulnerability Detection Tools with Java Cryptographic API Benchmarks

Several studies showed that misuses of cryptographic APIs are common in ...

Industrial Experience of Finding Cryptographic Vulnerabilities in Large-scale Codebases

Enterprise environments need to screen large-scale (millions of lines of...

Automatic Generation of Benchmarks for Plagiarism Detection Tools using Grammatical Evolution

This paper has been withdrawn by the authors due to a major rewriting....

BenchPress: Analyzing Android App Vulnerability Benchmark Suites

In recent years, various efforts have designed and developed benchmark s...

ToolCoder: Teach Code Generation Models to use API search tools

Automatically generating source code from natural language descriptions ...

PointEval: On the Impact of Pointer Analysis Frameworks

Pointer analysis is a foundational analysis leveraged by various static ...

Please sign up or login with your details

Forgot password? Click here to reset