Authors | M. Jørgensen, T. Halkjelsvik and K. Liestøl |
Title | When should we (not) use the mean magnitude of relative error (MMRE) as an error measure in software development effort estimation? |
Afilliation | Software Engineering |
Project(s) | Department of IT Management, EDOS: Effective Digitalization of Public Sector |
Status | Published |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Information and Software Technology |
Volume | 143 |
Pagination | 106784 |
Date Published | 03/2022 |
Publisher | Elsevier |
Abstract | Context: The mean magnitude of relative error (MMRE) is an error measure frequently used to evaluate and compare the estimation performance of prediction models and software professionals. Objective: This paper examines conditions for proper use of MMRE in effort estimation contexts. Method: We apply research on scoring functions to identify the type of estimates that minimizes the expected value of the MMRE. Results: We show that the MMRE is a proper error measure for estimates of the most likely (mode) effort, but not for estimates of the median or mean effort, provided that the effort usage is approximately log-normally distributed, which we argue is a reasonable assumption in many software development contexts. The relevance of the findings is demonstrated on real-world software development data. Conclusion: MMRE is not a proper measure of the accuracy of estimates of the median or mean effort, but may be used for the accuracy evaluation of estimates of most likely effort. |
Citation Key | 28156 |