|Authors||M. Jørgensen, T. Halkjelsvik and K. Liestøl|
|Title||When should we (not) use the mean magnitude of relative error (MMRE) as an error measure in software development effort estimation?|
|Project(s)||Department of IT Management, EDOS: Effective Digitalization of Public Sector|
|Publication Type||Journal Article|
|Year of Publication||2022|
|Journal||Information and Software Technology|
Context: The mean magnitude of relative error (MMRE) is an error measure frequently used to evaluate and compare the estimation performance of prediction models and software professionals.
Objective: This paper examines conditions for proper use of MMRE in effort estimation contexts.
Method: We apply research on scoring functions to identify the type of estimates that minimizes the expected value of the MMRE.
Results: We show that the MMRE is a proper error measure for estimates of the most likely (mode) effort, but not for estimates of the median or mean effort, provided that the effort usage is approximately log-normally distributed, which we argue is a reasonable assumption in many software development contexts. The relevance of the findings is demonstrated on real-world software development data.
Conclusion: MMRE is not a proper measure of the accuracy of estimates of the median or mean effort, but may be used for the accuracy evaluation of estimates of most likely effort.