4 ms·

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

by henning 2y ago

vasco 2y ago
And that's why runners don't track their race times.
- sam_lowry_ 2y ago
  Not all runners are competitive runners. I tack my time as a measure of progress among many other measurements, but my objective is to feel better and live a saner life. Metrics are harder on software engineers, the good ones delete code, the best ones make sure useless code never gets written in the first place. How do you measure that?
  - adchurch 2y ago
    > Metrics are harder on software engineers This is exactly the problem we're hoping to solve :') > the good ones delete code Our algorithm gives lots of credit for deleting old code! > the best ones make sure useless code never gets written in the first place This we don't really capture yet - but our hope is to continue along and give insights into this vital work as well down the line.
- ryandvm 2y ago
  If the metric is COINCIDENT with the goal, then it's not a problem. You can't get better at winning races without improving your race time (well... you can degrade everyone else's times I suppose). But if the metric is a PROXY for the goal, then the metric becomes the objective (not the actual goal). In this case, whatever this AI is measuring is going to be exactly what every dev drops into their Github Copilot prompt instructions.
  - dragonwriter 2y ago
    Making a metric an objective is an effective if often very costly way of testing the hypothesis that it is coincident rather than a proxy (which in cases more complex than racing is frequently a matter of dispute.)
adchurch 2y ago
This metric approximately measures functionality added, so the way to game it would be to add lots more functionality. This metric has no opinion on the nature of that functionality (i.e. it's not evaluating product decisions). So it doesn't tell the whole story, but it tells a much more accurate story than LOC or whatever other metrics people are using currently!
- henning 2y ago
  You are describing a feature factory where people ship features just to ship features. Shitloads of code gets shipped and yet there's no real forward momentum because none of it has any impact. It is a bad place to be.
  - adchurch 2y ago
    Wouldn't this be considered a very high performing engineering team with a very poor product team? We're only evaluating engineering - not the product decisions behind that engineering.
    - jrs235 2y ago
      Sure, a siloed organization. Some folks care about the whole process not just the work in front of them. In fact, one of the biggest demotivators and morale destroying things is doing work that isn't meaningful (and the person isn't just about making money for themselves which would make the work meaningful to them (some of us don't care to be mercenaries...).

As soon as people know how the metric is calculated, they will game that metric and it will cease to be useful.

vasco 2y ago

And that's why runners don't track their race times.

sam_lowry_ 2y ago
Not all runners are competitive runners. I tack my time as a measure of progress among many other measurements, but my objective is to feel better and live a saner life. Metrics are harder on software engineers, the good ones delete code, the best ones make sure useless code never gets written in the first place. How do you measure that?
- adchurch 2y ago
  > Metrics are harder on software engineers This is exactly the problem we're hoping to solve :') > the good ones delete code Our algorithm gives lots of credit for deleting old code! > the best ones make sure useless code never gets written in the first place This we don't really capture yet - but our hope is to continue along and give insights into this vital work as well down the line.
ryandvm 2y ago
If the metric is COINCIDENT with the goal, then it's not a problem. You can't get better at winning races without improving your race time (well... you can degrade everyone else's times I suppose). But if the metric is a PROXY for the goal, then the metric becomes the objective (not the actual goal). In this case, whatever this AI is measuring is going to be exactly what every dev drops into their Github Copilot prompt instructions.
- dragonwriter 2y ago
  Making a metric an objective is an effective if often very costly way of testing the hypothesis that it is coincident rather than a proxy (which in cases more complex than racing is frequently a matter of dispute.)

adchurch 2y ago

This metric approximately measures functionality added, so the way to game it would be to add lots more functionality. This metric has no opinion on the nature of that functionality (i.e. it's not evaluating product decisions). So it doesn't tell the whole story, but it tells a much more accurate story than LOC or whatever other metrics people are using currently!

henning 2y ago
You are describing a feature factory where people ship features just to ship features. Shitloads of code gets shipped and yet there's no real forward momentum because none of it has any impact. It is a bad place to be.
- adchurch 2y ago
  Wouldn't this be considered a very high performing engineering team with a very poor product team? We're only evaluating engineering - not the product decisions behind that engineering.
  - jrs235 2y ago
    Sure, a siloed organization. Some folks care about the whole process not just the work in front of them. In fact, one of the biggest demotivators and morale destroying things is doing work that isn't meaningful (and the person isn't just about making money for themselves which would make the work meaningful to them (some of us don't care to be mercenaries...).