What we do and don't know about software development effort estimation

From AcaWiki
Jump to: navigation, search

Citation: Magne Jorgensen (2014) What we do and don't know about software development effort estimation. IEEE Software (RSS)
DOI (original publisher): 10.1109/MS.2014.49
Semantic Scholar (metadata): 10.1109/MS.2014.49
Sci-Hub (fulltext): 10.1109/MS.2014.49
Internet Archive Scholar (search for fulltext): What we do and don't know about software development effort estimation
Wikidata (metadata): Q17784002
Download: http://www.infoq.com/articles/software-development-effort-estimation
Tagged: Computer Science (RSS)

Summary

Cost and effort overruns tend to be about 30% and haven't changed much from 1980s to now. Estimation methods haven't changed, expert estimation still dominates. But we know more; author notes 7 lessons supported by research:

  1. There Is No “Best” Effort Estimation Model or Method (important variable depends on context, also explains overfitting of advanced statistical estimation methods)
  2. Clients’ Focus on Low Price Is a Major Reason for Effort Overruns
  3. Minimum and Maximum Effort Intervals Are Too Narrow (estimates do not adequately reflect uncertainty)
  4. It’s Easy to Mislead Estimation Work and Hard to Recover from Being Misled (strongest when estimators aware of constraints such as budget, resulting in "estimation anchor" bias even if unintentional)
  5. Relevant Historical Data and Checklists Improve Estimation Accuracy
  6. Combining Independent Estimates Improves Estimation Accuracy (groupthink leading to more risk not found in software estimation research)
  7. Estimates Can Be Harmful (too low: low quality, too high: work expands; consider whether estimate is really needed)

3 estimation challenges research has no solution for:

  • How to Accurately Estimate the Effort of Mega - large, Complicated Software Projects (less relevant experience and data available, cf #5 above; large projects also involve complex interactions with stakeholders and organizational changes)
  • How to Measure Software Size and Complexity for Accurate Estimation
  • How to Measure and Predict Productivity (large difference among developers and team only discernible through trial; we don't even know if there are economies or diseconomies to scale for software production!)

Practices likely to improve estimation, quote:

  • "Develop and use simple estimation models tailored to local contexts in combination with expert estimation.
  • Use historical estimation error to set minimum - maximum effort intervals.
  • Avoid exposure to misleading and irrelevant estimation information.
  • Use checklists tailored to own organization.
  • Use structured, group - based estimation processes where independence of estimates are assured.
  • Avoid early estimates based on highly incomplete information."

Author provides "full set of empirical evidence I use to document the claims" in From Origami to Software Development: A Review of Studies on Judgment-Based Predictions of Performance Time.

Theoretical and Practical Relevance

Hacker News discussion including: "If I understand the paper, the only condition you need in order to arrive at accurate estimates is the absence of pressure to underestimate."

Projects with development work done by volunteers and/or distributed across organizations (commonly open source ones) tend to not estimate (is that true?) as nobody could be held to estimate anyway. Could this be an advantage?

Other than consequences (#7 above), estimation cost not mentioned. Is estimation itself significantly costly relative to subject of estimation?

Retrieved from "https://acawiki.org/index.php?title=What_we_do_and_don%27t_know_about_software_development_effort_estimation&oldid=10036"