AuthorsB. C. D. Anda, D. I. K. Sjøberg and A. Mockus
TitleVariability and Reproducibility in Software Engineering: a Study of Four Companies That Developed the Same System
AfilliationSoftware Engineering, Software Engineering
Publication TypeJournal Article
Year of Publication2009
JournalIEEE Transactions on Software Engineering

The scientific study of a phenomenon requires it to be reproducible. Mature engineering industries are recognized by projects and products that are, to some extent, reproducible. However, reproducibility in software engineering (SE) has not been thoroughly investigated, although generalizing the results of SE studies depend on SE phenomena being reproducible. We report a longitudinal multiple-case study of variations and reproducibility in software development, from bidding to deployment, on the basis of the same requirements specification. In a call for tender to 81 consultancy companies, 35 responded. Four of them developed the system independently. The firm price, planned schedule, and planned development process, had, respectively, “low”, “low”, and “medium” reproducibility. The contractor's internal costs, actual lead time, and schedule overrun of the projects had, respectively, “medium”, “high”, and “low” reproducibility. The quality dimensions of the delivered products, reliability, usability, and maintainability had, respectively, “low”, “high”, and “low” reproducibility. We proposed a coarse-grained model that uses software process inputs to predict key project outcomes. The comparisons of the model's predictions with the actual outcomes indicate some reproducibility. This initial work may contribute to developing more accurate models, but making SE more reproducible remains a great challenge for SE research, education, and industry.

Citation KeySimula.SE.195