AuthorsA. Arcuri and G. Fraser
TitleParameter Tuning Or Default Values? an Empirical Investigation in Search-Based Software Engineering
Afilliation, , Software Engineering
Project(s)The Certus Centre (SFI)
StatusPublished
Publication TypeJournal Article
Year of Publication2013
JournalEmpirical Software Engineering
Volume18
Number3
PublisherSpringer US
Abstract

Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No-Free-Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EVOSUITE tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that outperform by a large extent the “default” values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using “default” values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.

Citation KeySimula.simula.1164