What Predicts Software Developers' Productivity?

Citation: Emerson Murphy-Hill, Ciera Jaspan, Caitlin Sadowski, David Shepherd, Michael Phillips, Collin Winter, Andrea Knight, Edward Smith, Matthew Jorde (2019/02/19) What Predicts Software Developers' Productivity?. IEEE Transactions on Software Engineering (RSS)
DOI (original publisher): 10.1109/TSE.2019.2900308
Semantic Scholar (metadata): 10.1109/TSE.2019.2900308
Sci-Hub (fulltext): 10.1109/TSE.2019.2900308
Internet Archive Scholar (search for fulltext): What Predicts Software Developers' Productivity?
Download: https://doi.org/10.1109/TSE.2019.2900308
Tagged: Computer Science (RSS) software engineering (RSS)

Summary

The authors use survey developers on their productivity and other activities the authors believe might affect productivity in a variety of different companies. The authors have settled on a couple of factors they believe correlate the most with productivity. Of which, the most important are: job enthusiasm, peer support for new ideas, useful feedback about job performance.

Theoretical and Practical Relevance

This is one of the most complete and large-scale estimations of productivity factors. New research on developer productivity often starts from this study.

Research Questions

• What factors most strongly predict developers' self-rated productivity?
• How do these factors differ across companies?
• What predicts developer self-rated productivity, in particular, compared to other knowledge workers?

Companies Studied

Methodology

Questions

• Dependent factor: self-rating of "productivity"
  • Wanted the question to be a fixed-point of reference, with a high ceiling, that focuses on frequency and intensity of productivity.
  • Rated on Likert Scale "I regularly reach a high level of productivity."
• Independent factors
  • Subjective survey factors: based on survey questions from prior literature
  • Demographic factors: gender, tenure, seniority.
  • Attention question: to make sure people are reading the survey carefully "Respond with 'Somewhat disagree' to this item"
• Compare to software developers to other knowledge workers, to find unique differences. For "other knowledge workers," use Google data analysts.

Analysis

• Hold demographic factors constant for regression.
• Use Benjamini-Hochberg method to correct p-value over multiple linear models.
• For causality, look at how prior work relates the survey factors with productivity. This study itself does not attempt to establish that.

Results

• Significant positive correlation for both, CL merges more strongly, but there is still much unexplained variance.
• Google had a much higher response-rate than NI and ABB.
• Between 5 and 20% of the responses failed the attention question.

Demographic correlations

• Only at ABB was gender statistically significant.
  • Female respondants statistically significantly rated their productivity greater than their male counterparts, and custom genders reported greater than females.
• Tenure at position was statistically significant only at ABB.

Research Questions

• RQ1: What factors most strongly predict developers' self-rated productivity?
  • Job enthusiasm, peer support for new ideas, useful feedback about job performance
    • What makes software developers enthusiastic about their job? What accounts for differences in levels of enthusiasm between developers? What interventions can increase enthusiasm? This work can extend existing work on developer happiness [24] and motivation [25].
    • What kinds of new ideas are commonly expressed in software development practice? What actions influence developers' feelings of support for those ideas? What interventions can increase support for new ideas, while maintaining current commitments?
    • What kinds of job feedback do software engineers receive, and what makes it useful? What kinds of feedback is not useful? What interventions can increase the regularity and usefulness of feedback?
  • COCOMO factors were not as predictive. Either COCOMO is missing important factors or COCOMO measures something different than this kind of productivity.
• RQ2: How do these factors differ across companies?
  • Most consistent-between-companies: Use of remote work to concentrate, Useful feedback about job performance, Peer support for new ideas
    • These are all social/environemntal, not technical.
  • Most variant-betwee-companies: Use of best tools and practices, code reuse, accuracy of incoming information
    • Use of best tools/practices only matters in big codebases. Google has this, NI does not.
    • Code reuse is easier in a monorepo. Google has this, ABB does not.
    • Accurate information mattered more at NI than ABB. ABB has more support teams and layers at which to correct such information.
• RQ3: What predicts developer self-rated productivity, in particular, compared to other knowledge workers?
  • Analysts' productivity correlates uniquely with positive feelings about their teammates and time management autonomy.
  • Software developers' productivity correlates uniquely with doing a variety of tasks and working effectively way from their desk.

Future work

• Systematic literature review
• Consider more factors, such as demogrpahics
• Multiple dimensions and metrics of productivity
• Determine costs of changing factors which affect productivity.

Threats to validity

• They use self-rated productivity rather than objective metrics.
  • Prior literature seems to validate this choice.
• Their question had no time-window (productivity this year vs productivity today).
• They couldn't put too many questions, so they had to group things together. E.g. Q14 "Do you use best practices?", doesn't distinguish *which* best practices.
• They rely on prior work to establish causality and just measure correlation.
  • Maybe higher productivity causes higher enthusiasm, not the other way around.
• Maybe companies are not representative.
  • They tried to pick an archetype of each "kind" of company, new big tech (Google), old big tech (ABB), small tech (NI).
  • But the results are not weighted by prevalence of these kinds of companies. Maybe it is best to zoom in to the data which match your kind of company.
• Maybe analysts are not representative of other knowledge workers like lawyers.
• Maybe two factors are linked; no covariance analysis.
• Respondants might not be totally blind if they could guess the study's objectives from the survey questions.
• Had to reword the questions from software developers to suit analysts.

See Also

• Hackernews discussion