It is well known that there are productivity differences between different software developers. What is less well known, or sometimes less accepted, is the size of the difference in developer productivity.
Fredrick Brooks Jr in his 1975 book the ‘Mythical Man-Month’[1] observes there are significant differences in ability between different developers. He refers to research dating from the late 60s that shockingly suggests that if a group of experienced developers is compared there could be an order of magnitude difference between the best and the worse developers.
Software development practices have moved forward enormously since the 60s so it is interesting to ask whether there is still an order of magnitude variance between developers. It turns out there was a series of papers published around the turn of the century that looked at this question.
The first paper[2] in the series was published by Lutz Prechelt and looked at the productivity of C++ and Java developers. He asked participants to write a program to encode phone numbers as text strings in accordance with a requirement specification that he provided. Java and C++ remain in common use today and a program in either language to solve this task would look almost the same today as it did when the paper was written. Prechelt recorded the time participants spent implementing the program and he found the development time varied from 3 to 25 hours for C/C++ and from 4 to 63 hours for Java.
The participants were from a reasonably homogenous group as they were all Computer Science graduate students. They had an average 8 years programming experience and estimated that they had previously written on average 100K lines of code each.
It was stressed to the participants that they should focus on correctness and they were given financial incentives as an encouragement. Once their programs were submitted to the researchers, they were run through an automated acceptance test. As well as incorrect outputs, programs could be failed it they used too much memory or ran too slowly. In the event that a program failed, it was sent back to the author to be fixed and resubmitted. Guidance on why the program had failed was provided to help the author. Seventeen percent of the programs never passed the acceptance tests and as a result were excluded from the study.
The researchers did not require 100% accuracy in order to pass the acceptance tests and there was a large variation in accuracy between the programs that ‘passed’ the tests; as a result, 45% of the ‘passing’ programs contained bugs. But no correlation between development effort and accuracy was found. Hence the more productive developers did not achieve their speed at the expense of accuracy.
Even after the slowest programs were fixed or excluded, the run time speed of the programs varied enormously. For the C/C++ programs the ratio between the fastest and the slowest was 1200x and for Java the ratio was 74x. In the cloud, organisations pay for CPU on a per second basis so this performance difference would have on-going costs for the organisation. Again, no correlation was found between development effort and run time performance.
The researchers did find a strong correlation between code size and development effort. In general, the more lines of code in a program, the longer it takes to write and the more effort is needed down the road to maintain it. So not only had the most productive developers completed the task more quickly, they had written a program that would have lower on-going maintenance costs.
Conclusion
What is striking is that these differences in productivity are not between a heterogeneous group of developers, but a controlled group. Even, then there was a 20x difference in productivity. My personal experience is that these differences in productivity are also present in the real world.
The ‘very best’ developers who create correct, performant code in a fast time frame provide multiple financial saving to their employers; savings are made on development costs, maintenance costs and on cloud hosting charges.
The obvious conclusion is to staff your projects with small teams of the ‘very best’ developers. This is likely to be one reason that some start-ups have achieved enormous levels of productivity. Viaweb was sold to Yahoo for $49 million when it had just 21 employees[3]. This achievement was outdone by WhatsApp who at the time it was bought by Facebook for $22 billion had under 55 employees[4]!
But these developers are very rare and for the vast majority of organisations it is not realistic to recruit from such a small pool. Far more viable is to avoid individuals who are ‘0.1x developers’ working as software developers in your organisation[5]. There are many other roles such as test engineer, tech writer or scrum master where they might be more suitable.
There is a huge but unacknowledged difference in the abilities of developers. To be successful, projects need to take this on board both in planning and assigning tasks.
Picture entitled “Productivity” courtesy of Alpha Stock Images used under the CC BY SA 3.0 licence.
[1] Brooks, Frederick P, Jr. The Mythical Man-Month : Essays on Software Engineering. 1995. Page 30. [2] Prechelt, L. Technical Opinion: Comparing Java vs. C/C++ efficiency differences to inter-personal-differences. Communications of the ACM October 1999 (Vol.42, No. 10). [3] See https://www.nytimes.com/1998/06/09/business/company-news-yahoo-buying-viaweb-a-web-marketing-software-maker.html [4] See https://theconversation.com/whatsapp-bought-for-19-billion-what-do-its-employees-get-23496 [5] Sadowski, C. Zimmermann, T. Rethinking Productivity in Software Engineering. 2019. See chapter 1, The Mythical 10x Programmer by Lutz Prechelt
Comments