Support of Advanced Test cOverage Criteria for RObust and Secure Software
Software testing practices have experienced rapid changes in the recent years. This can be attributed to the wide adoption of agile methodologies (such as continuous integration and test-driven development), which put testing at the center of software development. Testing all the possible program inputs is practically intractable and thus the software testing community has defined coverage criteria (a.k.a. adequacy criteria or testing criteria ). These criteria are essential to an automated white-box testing process, as they are used by testing tools to guide the selection of new test cases, decide when testing should stop and assess the quality of a test suite. Dealing with advanced test criteria (such as boundary testing, MCDC, dafa-flow based criteria, etc.) is particularly important in safety critical applications. For instance, in regulated domains such as aeronautics, advanced coverage criteria are strict normative requirements that the tester must satisfy before delivering the software. In other domains, coverage criteria are recognized as a good practice for testing, and a key ingredient of test-driven development.
The existing testing tools support a limited number of test criteria in a hard-coded, non-generic manner. Therefore, in the current industrial practice, various tools are required to support different criteria. Moreover, in industry, in many cases test inputs are produced manually following informal criteria, based on the experience of test engineers in a particular domain. In this case such criteria are neither formalized nor supported by automated tools.
The SATOCROSS project (Support of Advanced Test cOverage Criteria for RObust and Secure Software) aims at improving software testing practice by bridging the gap between the potentialities offered by the huge body of academic work on (code-)coverage criteria on one side, and their limited tool support and use in the industry on the other side. The consortium includes two academic partners: CEA List (CEA) and the Interdisciplinary Centre for Security, Reliability and Trust (SnT) of Univ. of Luxembourg (UL), and one industrial partner MERCE (Mitsubishi Electric). The previous collaboration of the partners has convincingly demonstrated the benefits of the approach and proposed initial solutions for test generation and optimization for a small class of criteria. This collaboration helped to build an ambitious scientific program and motivated the submission of this proposal.
The partners will design generic and efficient testing techniques and tools for all major testing services: test input generation, detecting polluting test objectives and test assessment. They will focus on a large class of criteria important for the industrial practice. A particular attention will be paid to the continuous development paradigm. The project will feature these tools in an integrated open-source toolkit that will be evaluated on real-life industrial code provided by the industrial partner. Finally, the project will establish industrial application guidelines that will allow practitioners to benefit from the project results.
The project started in January 2019 and finished in December 2022. It is supported jointly by ANR and FNR agencies.