|Authors||D. Pradhan, S. Wang, T. Yue, S. Ali and M. Liaaen|
|Title||Search-Based Test Case Implantation for Testing Untested Configurations|
|Project(s)||The Certus Centre (SFI), Department of Engineering Complex Software Systems|
|Publication Type||Journal Article|
|Year of Publication||2019|
|Journal||Information and Software Technology|
|Keywords||genetic algorithms, Multi-objective optimization, Search, test case implantation|
Context: Modern large-scale software systems are highly configurable, and thus require a large number of test cases to be implemented and revised for testing a variety of system configurations. This makes testing highly configurable systems very expensive and time-consuming.
Objective: Driven by our industrial collaboration with a video conferencing company, we aim to automatically analyze and implant existing test cases (i.e., an original test suite) to test the untested configurations.
Method: We propose a search-based test case implantation approach (named as SBI) consisting of two key components: 1) Test case analyzer that statically analyzes each test case in the original test suite to obtain the program dependence graph for test case statements and 2) Test case implanter that uses multi-objective search to select suitable test cases for implantation using three operators, i.e., selection, crossover, and mutation (at the test suite level) and implants the selected test cases using a mutation operator at the test case level including three operations (i.e., addition, modification, and deletion).
Results: We empirically evaluated SBI with an industrial case study and an open source case study by comparing the implanted test suites produced by three variants of SBI with the original test suite using evaluation metrics such as statement coverage (SC), branch coverage (BC), and mutation score (MS). Results show that for both the case studies, the test suites implanted by the three variants of SBI performed significantly better than the original test suites. The best variant of SBI achieved on average 19.3% higher coverage of configuration variable values for both the case studies. Moreover, for the open source case study, the best variant of SBI managed to improve SC, BC, and MS with 5.0%, 7.9%, and 3.2%, respectively.
Conclusion: SBI can be applied to automatically implant a test suite with the aim of testing untested configurations and thus achieving higher configuration coverage.