AuthorsS. Ali, Y. Li, T. Yue and M. Zhang
TitleUncertainty-Wise and Time-Aware Test Case Prioritization with Multi-Objective Search
AfilliationSoftware Engineering
Publication TypeTechnical reports
Year of Publication2017
PublisherSimula Research Laboratory
KeywordsUncertainty-wise Testing; Test Case Prioritization; Multi-objective Search

Context: Complex systems (e.g., Cyber-Physical Systems) that interact with the real world, behave in an unstipulated manner while operating in uncertain environments. Testing such systems in uncertainty is a big challenge. Devising uncertainty-wise testing solutions can be considered as a mandate for dealing with this challenge. Though uncertainty-wise testing is gaining attention in the last few years, industry-strengthening solutions are still missing.
Objective: Our objective is to propose an uncertainty-wise test case prioritization approach that can significantly improve the cost-effectiveness of test case execution to maximize the occurrence of uncertainty.
Method: In this paper, we propose an uncertainty-wise, search-based, multi-objective test case prioritization approach, with a fitness function defined based on four cost-effectiveness measures: one subjective and one objective uncertainty measures, execution time, and transition coverage.
Results: We evaluated the well-known multi-objective search algorithm NSGA-II by comparing it with Greedy and Random Search (RS), with a real industrial case study. In addition, we created 72 additional simulated problems of varying complexity based on the real case study. Results show that NSGA-II achieved significantly better performance than RS and Greedy for both the real industrial case study and the simulated problems. On average, NSGA-II improved prioritization by 18% and 22% as compared to RS and Greedy respectively.
Conclusion: This paper presented an uncertainty-wise and time-aware test case prioritization, which was specifically developed to improve the cost and effectiveness of test case execution and at the same time maximizing the occurrence of uncertainties

Citation Key25015