7 ms·

To me the most interesting part of this is the claim that you can accurately and meaningfully measure software engineering productivity.

valval 2y ago
You can come up with measures for it and then watch them, that’s for sure.
- lr1970 2y ago
  when metric becomes the target it ceases to be a good metric. when discovered how it works developers will type the first character immediately after opening the log. edit: typo
  - joshuamorton 2y ago
    Only if the developer is being judged on the thing. If the tool is being judged on the thing, it's much less relevant. That is, I, personally, am not measured on how much AI generated code I create, and while the number is non-zero, I can't tell you what it is because I don't care and don't have any incentive to care. And I'm someone who is personally fairly bearish on the value of LLM-based codegen/autocomplete.
  - valval 2y ago
    That was my point, veiled in an attempt to be cute.
ozim 2y ago
You can - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev. For teams you can measure meaningful outcomes and improve team metrics. You shouldn’t really compare teams but it also is possible if you know what teams are doing. If you are some disconnected manager that thinks he can make decisions or improvements reducing things to single numbers - yeah that’s not possible.
- deely3 2y ago
  > For teams you can measure meaningful outcomes and improve team metrics. How? Which metrics?
  - ozim 2y ago
    That is what we pay managers -to figure out- for. They should find out which and how by knowing the team, familiarity with domain knowledge, understanding company dynamics, understanding customer, understanding market dynamics.
    - seanmcdirmid 2y ago
      That's basically a non-answer. Measuring "productivity" is a well known hard problem, and managers haven't really figured it out...
      - yorwba 2y ago
        Economists are generally fine with defining productivity as the ratio of aggregate outputs to aggregate inputs. Measuring it is not the hard part. The hard part is doing anything about it. If you can't attribute specific outputs to specific inputs, you don't know how to change inputs to maximize outputs. That's what managers need to do, but of course they're often just guessing.
        seanmcdirmid 2y ago
        Measuring human productivity is hard since we can't quantify output beyond silly metrics like lines of code written or amount of time speaking during meetings. Maybe if we were hunter/gatherers we could measure it by amount of animals killed.
        yorwba 2y ago
        That's why upthread we have https://news.ycombinator.com/item?id=41992562 https://news.ycombinator.com/item?id=41992562 "You can [accurately and meaningfully measure software engineering productivity] - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev." At the level of a company like Google, it's easy: both inputs and outputs are measured in terms of money.
UncleMeat 2y ago
At scale you can do this in a bunch of interesting ways. For example, you could measure "amount of time between opening a crash log and writing the first character of a new change" across 10,000s of engineers. Yes, each individual data point is highly messy. Alice might start coding as a means of investigation. Bob might like to think about the crash over dinner. Carol might get a really hard bug while David gets a really easy one. But at scale you can see how changes in the tools change this metric. None of this works to evaluate individuals or even teams. But it can be effective at evaluating tools.
- fwip 2y ago
  There's lots of stuff you can measure. It's not clear whether any of it is correlated with productivity. To use your example, a user with an LLM might say "LLM please fix this" as a first line of action, drastically improving this metric, even if it ruins your overall productivity.
zac23or 2y ago
I knew a superstar developer who worked on reports in an SQL tool. In the company metrics, the developer scored 420 points per month, the second developer scored 60 points. “Please learn how to score more points from the leader”, the boss would say. The superstar developer’s secret… he would send blank reports to clients (who would only realize it days later, and someone else would end up redoing the report), and he would score many more points without doing anything. I’ve seen this happen a lot in many different companies. As a friend of mine used to say, “it’s very rare, but it happens all the time.” I have no doubt that AI can help developers, but I don’t trust the metrics of the CEO or people who work on AI, because they are too involved in the subject.
- svieira 2y ago
  > When people are pressured to meet a target value there are three ways they can proceed: 1) They can work to improve the system 2) They can distort the system 3) Or they can distort the data https://commoncog.com/goodharts-law-not-useful/ https://commoncog.com/goodharts-law-not-useful/
- torginus 2y ago
  Honestly I doubt he got away with this for long (unless it was a very dysfunctional org). Being the best gets you noticed (in a good way), and screwing people over gets you noticed too (in a bad way), the combination of the two paints a target on your back.
  - GeoAtreides 2y ago
    > Being the best gets you noticed (in a good way), and screwing people over gets you noticed too (in a bad way), ah, to be young again...
    - torginus 2y ago
      I don't know what you're implying - I have had a few instances in my career when I went above and beyond and while I didn't receive too much praise for my efforts directly, after a while I noticed people who had no business knowing who I was, actually did. Now, I was really bad at capitalizing on it, so nothing much came of it, but still, there are some positive things that higher-ups do notice.

valval 2y ago

You can come up with measures for it and then watch them, that’s for sure.

lr1970 2y ago
when metric becomes the target it ceases to be a good metric. when discovered how it works developers will type the first character immediately after opening the log. edit: typo
- joshuamorton 2y ago
  Only if the developer is being judged on the thing. If the tool is being judged on the thing, it's much less relevant. That is, I, personally, am not measured on how much AI generated code I create, and while the number is non-zero, I can't tell you what it is because I don't care and don't have any incentive to care. And I'm someone who is personally fairly bearish on the value of LLM-based codegen/autocomplete.
- valval 2y ago
  That was my point, veiled in an attempt to be cute.

ozim 2y ago

You can - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev. For teams you can measure meaningful outcomes and improve team metrics. You shouldn’t really compare teams but it also is possible if you know what teams are doing. If you are some disconnected manager that thinks he can make decisions or improvements reducing things to single numbers - yeah that’s not possible.

deely3 2y ago
> For teams you can measure meaningful outcomes and improve team metrics. How? Which metrics?
- ozim 2y ago
  That is what we pay managers -to figure out- for. They should find out which and how by knowing the team, familiarity with domain knowledge, understanding company dynamics, understanding customer, understanding market dynamics.
  - seanmcdirmid 2y ago
    That's basically a non-answer. Measuring "productivity" is a well known hard problem, and managers haven't really figured it out...
    - yorwba 2y ago
      Economists are generally fine with defining productivity as the ratio of aggregate outputs to aggregate inputs. Measuring it is not the hard part. The hard part is doing anything about it. If you can't attribute specific outputs to specific inputs, you don't know how to change inputs to maximize outputs. That's what managers need to do, but of course they're often just guessing.
      - seanmcdirmid 2y ago
        Measuring human productivity is hard since we can't quantify output beyond silly metrics like lines of code written or amount of time speaking during meetings. Maybe if we were hunter/gatherers we could measure it by amount of animals killed.
        yorwba 2y ago
        That's why upthread we have https://news.ycombinator.com/item?id=41992562 https://news.ycombinator.com/item?id=41992562 "You can [accurately and meaningfully measure software engineering productivity] - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev." At the level of a company like Google, it's easy: both inputs and outputs are measured in terms of money.

UncleMeat 2y ago

At scale you can do this in a bunch of interesting ways. For example, you could measure "amount of time between opening a crash log and writing the first character of a new change" across 10,000s of engineers. Yes, each individual data point is highly messy. Alice might start coding as a means of investigation. Bob might like to think about the crash over dinner. Carol might get a really hard bug while David gets a really easy one. But at scale you can see how changes in the tools change this metric. None of this works to evaluate individuals or even teams. But it can be effective at evaluating tools.

fwip 2y ago
There's lots of stuff you can measure. It's not clear whether any of it is correlated with productivity. To use your example, a user with an LLM might say "LLM please fix this" as a first line of action, drastically improving this metric, even if it ruins your overall productivity.

zac23or 2y ago

I knew a superstar developer who worked on reports in an SQL tool. In the company metrics, the developer scored 420 points per month, the second developer scored 60 points. “Please learn how to score more points from the leader”, the boss would say. The superstar developer’s secret… he would send blank reports to clients (who would only realize it days later, and someone else would end up redoing the report), and he would score many more points without doing anything. I’ve seen this happen a lot in many different companies. As a friend of mine used to say, “it’s very rare, but it happens all the time.” I have no doubt that AI can help developers, but I don’t trust the metrics of the CEO or people who work on AI, because they are too involved in the subject.

svieira 2y ago
> When people are pressured to meet a target value there are three ways they can proceed: 1) They can work to improve the system 2) They can distort the system 3) Or they can distort the data https://commoncog.com/goodharts-law-not-useful/ https://commoncog.com/goodharts-law-not-useful/
torginus 2y ago
Honestly I doubt he got away with this for long (unless it was a very dysfunctional org). Being the best gets you noticed (in a good way), and screwing people over gets you noticed too (in a bad way), the combination of the two paints a target on your back.
- GeoAtreides 2y ago
  > Being the best gets you noticed (in a good way), and screwing people over gets you noticed too (in a bad way), ah, to be young again...
  - torginus 2y ago
    I don't know what you're implying - I have had a few instances in my career when I went above and beyond and while I didn't receive too much praise for my efforts directly, after a while I noticed people who had no business knowing who I was, actually did. Now, I was really bad at capitalizing on it, so nothing much came of it, but still, there are some positive things that higher-ups do notice.