Processing Fuzz Testing Results into an Evidence Report 2023-01-0039

In recent years, fuzz testing has established itself as a reliable and indispensable testing method for finding previously unknown and product specific vulnerabilities within the code base of automotive systems. As such, we see increased requirements for automotive products that call for fuzz testing per default. Based on the semidecidable characteristic for finding fuzz testing results, i.e., virtually an infinite test space, it is a non-trivial task to generate plausible evidence that sufficient fuzz testing has been applied to the target system. In this paper, starting from fuzz test result generation, we specify the individual steps necessary for preparing a sound evidence report. We describe how evidence is created in this context and which information is relevant. The traceability of fuzz testing product requirements is a driving factor thereby. We also analyze how useful Cybersecurity Assurance Levels (CAL) are in this process and how quantitative as well as qualitative metrics can be utilized as evidence. Finally, we discuss the conclusiveness of various fuzz testing evidence types and show the limits of evidence generation of this testing method. Our approach allows owners of security-relevant systems to provide evidence to customers or authorities showing that there was, based on a systematic risk-based methodology, sufficient fuzz testing conducted on the target system. Moreover, this approach brings the additional benefits of allowing more proven test diversification of the product, thereby reducing the effort on expensive manual testing methods like penetration testing and allowing more sophisticated overall test reporting of the product.