5 ms·

I'm not sure I follow. So you failed to measure software productivity in lines of code, therefore it follows that "There's No Such Thing as Software Productivit

by johnfn 2y ago

I'm not sure I follow. So you failed to measure software productivity in lines of code, therefore it follows that "There's No Such Thing as Software Productivity"? Don't you think that giving up after n=1 attempts at measuring software productivity might be a tad too fast to draw a generalized claim of impossibility? I might argue the real lesson learned is "Lines of Code are Not a Measure of Productivity in an Isolated, Toy Example". I suspect this sort of thing gets promulgated because it kind of massages our ego, like yes, they can measure other sorts of productivity, but not ours, oh no, we're too complex and intelligent, there's no way to measure the deep sorts of work that we do! Which, yes, OK, we're not exactly bricklayers, but surely, if you had to, you could do better.

strulovich 2y ago
Ok, but then what is a way to do it? The text gives an example to the core problem, and to argue differently requires thinking around it. In practice. I’ve seen many attempts at measuring productivity, but once you dig into them, you see they are just abstraction mechanisms above something that is similar to lines of code. I have yet to see an idea that sidesteps the core issue described in this post. Also, it applies to many types of work, and software is not unique in any way.
- nemomarx 2y ago
  profit generated I think is the high level one, and then you want to dig from there into how much the software development contributed to this.
  - nomel 2y ago
    That's very tricky thing to quantify, especially with "unsung heroes". If my work is in preventing problems, the guy that fixes problems will be seen as the one that contributes more to profit, since impact is directly observed/measured. This is something that one of the orgs I worked for eventually realized. The people f'ing up, and then fixing their mistakes, were the ones getting promotions/bonuses/raises, because they were the ones interacting with all the execs.
- bruce511 2y ago
  Happy Customers. All other measures are a proxy for happy customers. Actually, happy customers is also a proxy (the real measure is profits) but measuring profits directly (in the short term) can lead to decisions that have adverse long term effects. It's too easy to increase profits in the short term by avoiding long-term expenses. So, if you're in the business of software, the goal is happy customers. (And I use the word Customers carefully here. Not just Users who pay nothing, but Customers who spend money.) In a business context, it's really the only thing that matters. But, of course, it can be hard to measure (are they Happy?) and relies on multiple disciplines. Production (coding), Marketing, Sales, Support, Documentation, Training- all need to be working well to make it work. Ultimately if the big picture doesn't lead to Happy Customers (again, I stress, in a businesses context) then no-one is "being productive."
  - n4r9 2y ago
    Arguably the business is aiming for Paying Customers, not necessarily Happy ones :)
    - bruce511 2y ago
      I chose happy for a reason :) So the business is chasing profits. In the short term that means customers paying money - any will do (happy or unhappy). But in the long term, happy is the key. Happy customers are the single biggest marketing tool you have. Happy customers promote and recommend you. Unhappy customers do the opposite (and are more effective at doing so.) So, if the metric stops at Customers then you are greatly missing the long-term value. Since a good business is planning for the long term, not just right now, Happy customers I the correct metric. Remember, you ultimately get what you measure (no more).
      - Cthulhu_ 2y ago
        > Happy customers are the single biggest marketing tool you have. Happy customers promote and recommend you. Unhappy customers do the opposite (and are more effective at doing so.) And yet, there's some irrational counter examples; there's video games that has huge detractors while having huge financial success. Negative reviews on Steam, "4000 hours played". The metrics say they aren't happy with the product... but they still play it, talk about it, may have pulled in friends to play it, and spend money on it. People can be unhappy about a product but still pay and promote it, counter-intuitively. Of course, the unhappiness is what they say, their behaviour says otherwise so for the sake of the metrics they would be considered happy I suppose?
- foobiekr 2y ago
  Revenue per employee $ spent.
- eacapeisfutuile 2y ago
  As already mentioned, people measure value return by revenue gain. It is irrelevant to attribute it to some construct like a line of code.
- dfabulich 2y ago
  I recommend measuring job satisfaction instead of developer productivity. It's the "least bad" proxy metric I know of. https://redfin.engineering/measure-job-satisfaction-instead-of-software-engineering-productivity-418779ce3451 https://redfin.engineering/measure-job-satisfaction-instead-...
  - deleted 2y ago
    [deleted]
  - musicale 2y ago
    The interesting corollary to this approach seems to be that productivity barriers are largely external. A potential riske seems to be feedback systems where job satisfaction is determined by high or low pay.
n_ary 2y ago
> it kind of massages our ego, like yes, they can measure other sorts of productivity, but not ours, oh no, we're too complex and intelligent, there's no way to measure the deep sorts of work that we do! Which, yes, OK, we're not exactly bricklayers, but surely, if you had to, you could do better. Productivity is not a cut-n-dry stat for any kind of knowledge work. For an admin, we could say they were more productive if they handled more cases than last month. But this does not account for tricky cases or ambiguous cases, which might have taken 2x time to sort out but on rote numbers it looks they were less productive. For us, there are no specific goals(except arbitrary imaginary deadlines by clueless management and sprint points) to correctly measure productivity. A junior engineer may spend 4h each day blasting out many lines of codes, a mid/senior may spend more hours in reviews or meetings. A 5 line PR with test coverage is faster to review than a 800+ LOC PR with additional tests and have significant risk of breaking something, so review number is also not a good indicator. I need to coordinate with min. 12 people to get any new credentials or access to specific env owned by another team, which is not countable as productive but unavoidable. How about that garbage meeting where I was just spending time yawning because some new tech lead who believes that a modular monolith system is better if it was rewritten as event based. Productivity measurement is hard. To take your envy(I sense it on the quoted remark) to another place, think about how you go about measuring productivity of a CEO, PO, PM, Scrum Master, Agilist/Agile-Specialist etc?
- johnfn 2y ago
  > To take your envy(I sense it on the quoted remark) to another place, think about how you go about measuring productivity of a CEO, PO, PM, Scrum Master, Agilist/Agile-Specialist etc? I have no envy; I'm a software engineer as well. I just don't seem to struggle, as others claim to, in measuring productivity. I find it fairly straightforward to see that some people are more productive than others, at least in my workplace where we can hold most things constant (e.g. meeting count, managerial ability, etc, etc). Yes, there may be external factors, and that's unfortunate. Yes, it's not fair that some people are more productive than others despite putting in half the effort. But I'm not going to stick my head in the sand and pretend it's invisible.
  - 2y ago
WgaqPdNr7PGLGVW 2y ago
> Don't you think that giving up after n=1 attempts at measuring software productivity might be a tad too fast to draw a generalized claim of impossibility? Software developers should look at anyone claiming they can measure software productivity as a snake oil salesman. We have seen hundreds of attempts over the years and they have all "failed". More accurately they all have large error bars and biases. Researchers can and should continue looking into how to measure software development productivity. It is likely over the next few decades we will start to understand how to measure it appropriately.
EarthBlues 2y ago
I think people get hung up on wanting to collapse developer productivity into a single dimension, usually for stack-ranking purposes. This, I think, is always going to punish good engineers and reward bad ones to some degree. Measuring developer productivity should, in my opinion, have one dimension for speed, one for quality, and one for user impact. LOC can be fine as a measurement for speed, you just don’t want to look at it in isolation. You would want to also measure, for example, escape rate and usage for the features the developer worked on, and be willing to change or refine these if circumstances require it. You also need to look for different profiles based on the developer’s level of seniority. A senior dev probably shouldn’t be writing as much code as a contributor, but their user impact should be high, and escape rate low. Analyzing differences between teams is important, as well. A team that has a lot of escapes or little user impact probably has issues that need management attention, and may not have anything at all to do with individual developer productivity or ability. In brief, the numbers are there to help you make better management decisions, not to relieve you of having to make them.
nradov 2y ago
Function point analysis works well enough for measuring software productivity in most domains, provided that quality is held constant. Like other software productivity metrics it's only meaningful at the level of a complete product team and worse than useless for evaluating individual team members. https://doi.ieeecomputersociety.org/10.1109/32.44364 https://doi.ieeecomputersociety.org/10.1109/32.44364 So you can measure productivity with a reasonable level of accuracy and consistency, but then what? For most organizations it's not actionable so calculating that metric ends up as another form of waste.
- robertlagrant 2y ago
  > Function point analysis works well enough for measuring software productivity in most domains, provided that quality is held constant Well enough for what? And how do you measure quality?
michaelmrose 2y ago
Bricklayers have measures that are usable by people who know little to nothing about the profession who are incapable of doing the work. This is what people want. An objective measure that can be applied by people functionally incapable of actually doing the work. Wherein the work isn't repeating existing work no such measure should be expected to exist. Any creative work is going to suffer from the same problem. If you asked a brick layer to develop a better workflow for his fellows to work more effectively in a particular sitiation it would be the same. Are you going to measure words per minute?
- gregors 2y ago
  The problem is that corporate types don't see programming as creative work. Type faster!!!! If you push the buttons faster the product gets build faster! Also imagine every other week as the bricklayer had half a wall built a PM comes out and says, "tear this all down and move it 3 feet to the north", and then the week after, "now move it east 1 foot, and why is it taking you so long?"
fumeux_fume 2y ago
You certainly did fail to see the idea being conveyed by a simple, but poignant anecdote. If this is all you have to say, I might argue the real ego being massaged is your own; by yourself.
rob74 2y ago
This is not an isolated toy example, it's reductio ad absurdum (https://en.wikipedia.org/wiki/Reductio_ad_absurdum https://en.wikipedia.org/wiki/Reductio_ad_absurdum) - and until today I haven't seen a way to measure software productivity (might also apply to other knowledge work) that is resistant to that...
JimDabell 2y ago
I agree. I know from looking at a story that if Alice picks it up it’ll take her about a day and be decent quality, but if Bob picks it up it’ll take him about a week and be worse quality. And I know that this will be pretty consistent week-in, week-out. Developer productivity exists and I am totally comfortable describing Alice as being more productive than Bob. The fact that there isn’t a good, generic way to distill this into particular values does not mean that the phenomenon doesn’t exist. We’ve all worked with Alice and Bobs. Claiming there’s no such thing as developer productivity is denying what we’ve all experienced. We aren’t special snowflakes whose work is beyond the ken of mortal men. Our work has value and sometimes we produce a lot of value in a particular time period and sometimes we product a little. This is productivity.
perrygeo 2y ago
> "Lines of Code are Not a Measure of Productivity in an Isolated, Toy Example" Calling this phenomenon a toy example is a bit out of touch. I've seen this every single day of my 25 year career. Value is produced by solving problems, not writing code. The solution with fewer lines of code, fewer new abstractions, lower complexity (yes, complexity IS objective and quantifiable) is invariably the best solution. Solutions that involve no code at all are the gold standard. Adding/subtracting the right code produces value. But adding the wrong code decreases value. The logic of "productivity as code" contains a fatal flaw - it ignores this and assumes all code is right. Reality: everything hinges on the right vs wrong distinction which is inherently subjective. Code is too often a net liability - you must account for the risk! By contrast, the "productivity as solving problems" approach is entirely objective. You can observe the problem, test hypotheses about the cause, and measure the situation after the intervention. A well stated problem has no ambiguity. I don't see any support whatsover for the "productivity as code" idea. It's empirically false and lacks logical consistency.
feoren 2y ago
> they can measure other sorts of productivity Spoiler: they can't do that either. How do you measure the productivity of a bridge engineer designing a new suspension bridge? Number of struts placed on the blueprint? Even with more manual labor, it's very hard. Who's the most productive out of these three: 1. A carpenter who builds a house from a blueprint in 6 weeks. It collapses after 3 years. 2. A carpenter who builds a house from the same blueprint in 12 weeks. It stays up, but has visible defects that affect its value. 3. A carpenter who builds a house from the same blueprint in 20 weeks. It stays up for over a century and eventually becomes a historical landmark due to its lasting beauty and careful construction. Which one was more productive?

by johnfn 2y ago

strulovich 2y ago

Ok, but then what is a way to do it? The text gives an example to the core problem, and to argue differently requires thinking around it. In practice. I’ve seen many attempts at measuring productivity, but once you dig into them, you see they are just abstraction mechanisms above something that is similar to lines of code. I have yet to see an idea that sidesteps the core issue described in this post. Also, it applies to many types of work, and software is not unique in any way.

nemomarx 2y ago
profit generated I think is the high level one, and then you want to dig from there into how much the software development contributed to this.
- nomel 2y ago
  That's very tricky thing to quantify, especially with "unsung heroes". If my work is in preventing problems, the guy that fixes problems will be seen as the one that contributes more to profit, since impact is directly observed/measured. This is something that one of the orgs I worked for eventually realized. The people f'ing up, and then fixing their mistakes, were the ones getting promotions/bonuses/raises, because they were the ones interacting with all the execs.
bruce511 2y ago
Happy Customers. All other measures are a proxy for happy customers. Actually, happy customers is also a proxy (the real measure is profits) but measuring profits directly (in the short term) can lead to decisions that have adverse long term effects. It's too easy to increase profits in the short term by avoiding long-term expenses. So, if you're in the business of software, the goal is happy customers. (And I use the word Customers carefully here. Not just Users who pay nothing, but Customers who spend money.) In a business context, it's really the only thing that matters. But, of course, it can be hard to measure (are they Happy?) and relies on multiple disciplines. Production (coding), Marketing, Sales, Support, Documentation, Training- all need to be working well to make it work. Ultimately if the big picture doesn't lead to Happy Customers (again, I stress, in a businesses context) then no-one is "being productive."
- n4r9 2y ago
  Arguably the business is aiming for Paying Customers, not necessarily Happy ones :)
  - bruce511 2y ago
    I chose happy for a reason :) So the business is chasing profits. In the short term that means customers paying money - any will do (happy or unhappy). But in the long term, happy is the key. Happy customers are the single biggest marketing tool you have. Happy customers promote and recommend you. Unhappy customers do the opposite (and are more effective at doing so.) So, if the metric stops at Customers then you are greatly missing the long-term value. Since a good business is planning for the long term, not just right now, Happy customers I the correct metric. Remember, you ultimately get what you measure (no more).
    - Cthulhu_ 2y ago
      > Happy customers are the single biggest marketing tool you have. Happy customers promote and recommend you. Unhappy customers do the opposite (and are more effective at doing so.) And yet, there's some irrational counter examples; there's video games that has huge detractors while having huge financial success. Negative reviews on Steam, "4000 hours played". The metrics say they aren't happy with the product... but they still play it, talk about it, may have pulled in friends to play it, and spend money on it. People can be unhappy about a product but still pay and promote it, counter-intuitively. Of course, the unhappiness is what they say, their behaviour says otherwise so for the sake of the metrics they would be considered happy I suppose?
foobiekr 2y ago
Revenue per employee $ spent.
eacapeisfutuile 2y ago
As already mentioned, people measure value return by revenue gain. It is irrelevant to attribute it to some construct like a line of code.
dfabulich 2y ago
I recommend measuring job satisfaction instead of developer productivity. It's the "least bad" proxy metric I know of. https://redfin.engineering/measure-job-satisfaction-instead-of-software-engineering-productivity-418779ce3451 https://redfin.engineering/measure-job-satisfaction-instead-...
- deleted 2y ago
  [deleted]
- musicale 2y ago
  The interesting corollary to this approach seems to be that productivity barriers are largely external. A potential riske seems to be feedback systems where job satisfaction is determined by high or low pay.

n_ary 2y ago

> it kind of massages our ego, like yes, they can measure other sorts of productivity, but not ours, oh no, we're too complex and intelligent, there's no way to measure the deep sorts of work that we do! Which, yes, OK, we're not exactly bricklayers, but surely, if you had to, you could do better. Productivity is not a cut-n-dry stat for any kind of knowledge work. For an admin, we could say they were more productive if they handled more cases than last month. But this does not account for tricky cases or ambiguous cases, which might have taken 2x time to sort out but on rote numbers it looks they were less productive. For us, there are no specific goals(except arbitrary imaginary deadlines by clueless management and sprint points) to correctly measure productivity. A junior engineer may spend 4h each day blasting out many lines of codes, a mid/senior may spend more hours in reviews or meetings. A 5 line PR with test coverage is faster to review than a 800+ LOC PR with additional tests and have significant risk of breaking something, so review number is also not a good indicator. I need to coordinate with min. 12 people to get any new credentials or access to specific env owned by another team, which is not countable as productive but unavoidable. How about that garbage meeting where I was just spending time yawning because some new tech lead who believes that a modular monolith system is better if it was rewritten as event based. Productivity measurement is hard. To take your envy(I sense it on the quoted remark) to another place, think about how you go about measuring productivity of a CEO, PO, PM, Scrum Master, Agilist/Agile-Specialist etc?

johnfn 2y ago
> To take your envy(I sense it on the quoted remark) to another place, think about how you go about measuring productivity of a CEO, PO, PM, Scrum Master, Agilist/Agile-Specialist etc? I have no envy; I'm a software engineer as well. I just don't seem to struggle, as others claim to, in measuring productivity. I find it fairly straightforward to see that some people are more productive than others, at least in my workplace where we can hold most things constant (e.g. meeting count, managerial ability, etc, etc). Yes, there may be external factors, and that's unfortunate. Yes, it's not fair that some people are more productive than others despite putting in half the effort. But I'm not going to stick my head in the sand and pretend it's invisible.
- 2y ago

WgaqPdNr7PGLGVW 2y ago

> Don't you think that giving up after n=1 attempts at measuring software productivity might be a tad too fast to draw a generalized claim of impossibility? Software developers should look at anyone claiming they can measure software productivity as a snake oil salesman. We have seen hundreds of attempts over the years and they have all "failed". More accurately they all have large error bars and biases. Researchers can and should continue looking into how to measure software development productivity. It is likely over the next few decades we will start to understand how to measure it appropriately.

EarthBlues 2y ago

I think people get hung up on wanting to collapse developer productivity into a single dimension, usually for stack-ranking purposes. This, I think, is always going to punish good engineers and reward bad ones to some degree. Measuring developer productivity should, in my opinion, have one dimension for speed, one for quality, and one for user impact. LOC can be fine as a measurement for speed, you just don’t want to look at it in isolation. You would want to also measure, for example, escape rate and usage for the features the developer worked on, and be willing to change or refine these if circumstances require it. You also need to look for different profiles based on the developer’s level of seniority. A senior dev probably shouldn’t be writing as much code as a contributor, but their user impact should be high, and escape rate low. Analyzing differences between teams is important, as well. A team that has a lot of escapes or little user impact probably has issues that need management attention, and may not have anything at all to do with individual developer productivity or ability. In brief, the numbers are there to help you make better management decisions, not to relieve you of having to make them.

nradov 2y ago

Function point analysis works well enough for measuring software productivity in most domains, provided that quality is held constant. Like other software productivity metrics it's only meaningful at the level of a complete product team and worse than useless for evaluating individual team members. https://doi.ieeecomputersociety.org/10.1109/32.44364 https://doi.ieeecomputersociety.org/10.1109/32.44364 So you can measure productivity with a reasonable level of accuracy and consistency, but then what? For most organizations it's not actionable so calculating that metric ends up as another form of waste.

robertlagrant 2y ago
> Function point analysis works well enough for measuring software productivity in most domains, provided that quality is held constant Well enough for what? And how do you measure quality?

michaelmrose 2y ago

Bricklayers have measures that are usable by people who know little to nothing about the profession who are incapable of doing the work. This is what people want. An objective measure that can be applied by people functionally incapable of actually doing the work. Wherein the work isn't repeating existing work no such measure should be expected to exist. Any creative work is going to suffer from the same problem. If you asked a brick layer to develop a better workflow for his fellows to work more effectively in a particular sitiation it would be the same. Are you going to measure words per minute?

gregors 2y ago
The problem is that corporate types don't see programming as creative work. Type faster!!!! If you push the buttons faster the product gets build faster! Also imagine every other week as the bricklayer had half a wall built a PM comes out and says, "tear this all down and move it 3 feet to the north", and then the week after, "now move it east 1 foot, and why is it taking you so long?"

fumeux_fume 2y ago

You certainly did fail to see the idea being conveyed by a simple, but poignant anecdote. If this is all you have to say, I might argue the real ego being massaged is your own; by yourself.

rob74 2y ago

This is not an isolated toy example, it's reductio ad absurdum (https://en.wikipedia.org/wiki/Reductio_ad_absurdum https://en.wikipedia.org/wiki/Reductio_ad_absurdum) - and until today I haven't seen a way to measure software productivity (might also apply to other knowledge work) that is resistant to that...

JimDabell 2y ago

I agree. I know from looking at a story that if Alice picks it up it’ll take her about a day and be decent quality, but if Bob picks it up it’ll take him about a week and be worse quality. And I know that this will be pretty consistent week-in, week-out. Developer productivity exists and I am totally comfortable describing Alice as being more productive than Bob. The fact that there isn’t a good, generic way to distill this into particular values does not mean that the phenomenon doesn’t exist. We’ve all worked with Alice and Bobs. Claiming there’s no such thing as developer productivity is denying what we’ve all experienced. We aren’t special snowflakes whose work is beyond the ken of mortal men. Our work has value and sometimes we produce a lot of value in a particular time period and sometimes we product a little. This is productivity.

perrygeo 2y ago

> "Lines of Code are Not a Measure of Productivity in an Isolated, Toy Example" Calling this phenomenon a toy example is a bit out of touch. I've seen this every single day of my 25 year career. Value is produced by solving problems, not writing code. The solution with fewer lines of code, fewer new abstractions, lower complexity (yes, complexity IS objective and quantifiable) is invariably the best solution. Solutions that involve no code at all are the gold standard. Adding/subtracting the right code produces value. But adding the wrong code decreases value. The logic of "productivity as code" contains a fatal flaw - it ignores this and assumes all code is right. Reality: everything hinges on the right vs wrong distinction which is inherently subjective. Code is too often a net liability - you must account for the risk! By contrast, the "productivity as solving problems" approach is entirely objective. You can observe the problem, test hypotheses about the cause, and measure the situation after the intervention. A well stated problem has no ambiguity. I don't see any support whatsover for the "productivity as code" idea. It's empirically false and lacks logical consistency.

feoren 2y ago

> they can measure other sorts of productivity Spoiler: they can't do that either. How do you measure the productivity of a bridge engineer designing a new suspension bridge? Number of struts placed on the blueprint? Even with more manual labor, it's very hard. Who's the most productive out of these three: 1. A carpenter who builds a house from a blueprint in 6 weeks. It collapses after 3 years. 2. A carpenter who builds a house from the same blueprint in 12 weeks. It stays up, but has visible defects that affect its value. 3. A carpenter who builds a house from the same blueprint in 20 weeks. It stays up for over a century and eventually becomes a historical landmark due to its lasting beauty and careful construction. Which one was more productive?