|Authors||B. C. D. Anda, D. I. K. Sjøberg and A. Mockus|
|Title||Variability and Reproducibility in Software Engineering: a Study of Four Companies That Developed the Same System|
|Afilliation||Software Engineering, Software Engineering|
|Publication Type||Journal Article|
|Year of Publication||2009|
|Journal||IEEE Transactions on Software Engineering|
The scientific study of a phenomenon requires it to be reproducible. Mature engineering industries are recognized by projects and products that are, to some extent, reproducible. However, reproducibility in software engineering (SE) has not been thoroughly investigated, although generalizing the results of SE studies depend on SE phenomena being reproducible. We report a longitudinal multiple-case study of variations and reproducibility in software development, from bidding to deployment, on the basis of the same requirements specification. In a call for tender to 81 consultancy companies, 35 responded. Four of them developed the system independently. The firm price, planned schedule, and planned development process, had, respectively, “low”, “low”, and “medium” reproducibility. The contractor's internal costs, actual lead time, and schedule overrun of the projects had, respectively, “medium”, “high”, and “low” reproducibility. The quality dimensions of the delivered products, reliability, usability, and maintainability had, respectively, “low”, “high”, and “low” reproducibility. We proposed a coarse-grained model that uses software process inputs to predict key project outcomes. The comparisons of the model's predictions with the actual outcomes indicate some reproducibility. This initial work may contribute to developing more accurate models, but making SE more reproducible remains a great challenge for SE research, education, and industry.