AuthorsA. Arcuri and G. Fraser
EditorsM. B. Cohen and M. Ó. Cinnéide
TitleOn Parameter Tuning in Search Based Software Engineering
AfilliationSoftware Engineering, Software Engineering
Publication TypeProceedings, refereed
Year of Publication2011
Conference Namenternational Symposium on Search Based Software Engineering (SSBSE)
ISBN Number10.1007/978-3-642-23716-4

When applying search-based software engineering (SBSE) techniques one is confronted with a multitude of different parameters that need to be chosen: Which population size for a genetic algorithm? Which selection mechanism to use? What settings to use for dozens of other parameters? This problem not only troubles users who want to apply SBSE tools in practice, but also researchers performing experimentation - how to compare algorithms that can have different parameter settings? To shed light on the problem of parameters, we performed the largest empirical analysis on parameter tuning in SBSE to date, collecting and statistically analysing data from more than a million experiments. As case study, we chose test data generation, one of the most popular problems in SBSE. Our data confirm that tuning does have a critical impact on algorithmic performance, and over-fitting of parameter tuning is a dire threat to external validity of empirical analyses in SBSE. Based on this large empirical evidence, we give guidelines on how to handle parameter tuning.

Citation KeySimula.simula.678