AuthorsA. Arcuri and L. C. Briand
TitleA Hitchhiker's Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering
AfilliationSoftware Engineering, The Certus Centre (SFI), Software Engineering
Project(s)The Certus Centre (SFI)
StatusPublished
Publication TypeJournal Article
Year of Publication2012
JournalSoftware Testing, Verification and Reliability
Volume24
Issue3
Pagination219-250
Date Published05/2014
PublisherJohn Wiley & Sons
Abstract

Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation. However, randomized algorithms are affected by chance, and so require the use of appropriate statistical tests to be properly analyzed in a sound manner. This paper features a systematic review regarding recent publications in 2009 and 2010 showing that, overall, empirical analyses involving randomized algorithms in software engineering tend to not properly account for the random nature of these algorithms. Many of the novel techniques presented clearly appear promising, but the lack of soundness in their empirical evaluations casts unfortunate doubts on their actual usefulness. In software engineering, though there are guidelines on how to carry out empirical analyses involving human subjects, those guidelines are not directly and fully applicable to randomized algorithms. Furthermore, many of the text books on statistical analysis are written from the viewpoints of social and natural sciences, which present different challenges from randomized algorithms. To address the questionable overall quality of the empirical analyses reported in the systematic review, this paper provides guidelines on how to carry out and properly analyze randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing which is by far the most frequent application area of randomized algorithms within software engineering.

DOI10.1002/stvr.1486
Citation KeySimula.simula.1564