3 ms·

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterex

While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples: * In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it. * In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful * In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving * In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.

mnowicki 7y ago
Please stick with me here, probably going to be a super long post but I'm sharing my thoughts on the whole topic end-to-end with the goal of addressing the root problem here, I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired outcome and attempt to implement a system that teams can use to apply the metric to their work. It ends up being effectively impossibly complex to create the formula, and even if you could create it, it would take longer it than it would to write the software that it's supposed to measure. What we're left with is 'the best that we can do', and that's what we have been doing. We absolutely COULD make more accurate(yet still very imperfect) metrics that more closely model the actual goals of a project by simply including more factors(bugs, readability of code, difficulty of feature, size of feature/LOC, number of modules/classes that the code spans or communicates with in some way, how easy-to-read and easy-to-modify the existing code that needs to be altered is, etc times 1,000,000). But adding more factors just makes the metric exponentially more complex and difficult to implement. I'm sure people have tried using way more fine-grained metrics than what we have now - but nobody knows about them because nobody uses them anymore because they weren't worth the time and effort. So we're kind of left with what we have, which in my opinion as a software engineer is definitely not good enough. Which is pretty bad, because by 'good enough' I just mean better than doing nothing at all. The only problem here that can be solved, and therefore that is worth solving, is why these metrics are still commonly enforced despite being counterproductive. Obviously managers managers-of-managers feel a need to have some nice, concrete, objective numbers that they can look at to see how well developers and teams are doing. The people pushing these metrics are often the ones who have no other way of understanding what exactly developers are doing, either because they've never written software before or because they are too far removed from the level of the guys actually writing code to pay attention to what they're actually doing beyond just looking at a single number. It makes sense for managers who don't know what engineers are doing to seek out a different way to measure their performance and progress. The solution isn't do nothing, and it's not story points or any of that BS. But what are the alternatives? The only one I can think of is, don't put a manager in a position to manage individual developers if he/she is unable to understand what those individuals are even doing. Having an experienced senior engineer with the right skillset for management directly in charge of every team is amazingly valuable in my opinion. It's also vital that those higher up on the ladder are able to trust these manager-engineers, who are basically acting as an abstraction layer between the regular engineers and higher management. A lot of teams already have this in the form of a senior engineer or software architect, my team has one and it works out amazingly. We all feel that doing a good job will be recognized and rewarded, problems are easily identified and addressed, we get things done very quickly, everyone gets along, and there's not a lot of pressure. This is just my own single anecdotal experience and it has worked out very well but I'm aware it may not always be that way, so I'm interested in hearing others thoughts/experiences/solutions
- Fire-Dragon-DoL 7y ago
  Just to highlight this, this is what Toyota does with its engineers (not limited to software). Managers are engineers of towering technical competence.
iudqnolq 7y ago
I think a better version is "you get what you measure". Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n. (I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/ https://xkcd.com/810/)
plorkyeran 7y ago
If you reward a team for increasing story points completed, then you will very quickly find that every single story has the maximum possible estimate. In the longer term you'd get stories split into more smaller stories far more often than is reasonable. It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.
- ratww 7y ago
  I saw that happen in a team whose OKR was stupidly tied to story points. They went from 20 points per sprint to 70 overnight, by splitting stories as most as possible. It was hilarious when they reached their quarter OKR in less than a month.
  - asplake 7y ago
    I have heard this described as “quantitative easing” for story points, and yes, hilarious.
jghn 7y ago
It wasn’t that long ago that OPS was not rewarded, and batting average was king. It took a lot of energy to change that tide and reward players for focusing on other metrics. My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.
ben509 7y ago
I think your counterexamples reveal why those metrics work: * In baseball, batting average and OPS require you to compete against other teams. * In finance, ROI is tested in the marketplace. * In sales, conversions require you to convince a skeptical customer. The software metrics are all things we have fairly direct control over. Maybe that suggests an amendment to Godhart's law: > When a measure that is controlled by the measuree becomes a target, it ceases to be a good measure.
- mnowicki 7y ago
  I was just writing a post that said the same exact thing with about 12 times as many words. Good thing I read your comment before I wasted another 20 minutes writing the other half of my response
chiefalchemist 7y ago
Maybe. But are you going to deduct for bugs? Or other flaws due to the metric not being "done right, and complete"?

mnowicki 7y ago

Please stick with me here, probably going to be a super long post but I'm sharing my thoughts on the whole topic end-to-end with the goal of addressing the root problem here, I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired outcome and attempt to implement a system that teams can use to apply the metric to their work. It ends up being effectively impossibly complex to create the formula, and even if you could create it, it would take longer it than it would to write the software that it's supposed to measure. What we're left with is 'the best that we can do', and that's what we have been doing. We absolutely COULD make more accurate(yet still very imperfect) metrics that more closely model the actual goals of a project by simply including more factors(bugs, readability of code, difficulty of feature, size of feature/LOC, number of modules/classes that the code spans or communicates with in some way, how easy-to-read and easy-to-modify the existing code that needs to be altered is, etc times 1,000,000). But adding more factors just makes the metric exponentially more complex and difficult to implement. I'm sure people have tried using way more fine-grained metrics than what we have now - but nobody knows about them because nobody uses them anymore because they weren't worth the time and effort. So we're kind of left with what we have, which in my opinion as a software engineer is definitely not good enough. Which is pretty bad, because by 'good enough' I just mean better than doing nothing at all. The only problem here that can be solved, and therefore that is worth solving, is why these metrics are still commonly enforced despite being counterproductive. Obviously managers managers-of-managers feel a need to have some nice, concrete, objective numbers that they can look at to see how well developers and teams are doing. The people pushing these metrics are often the ones who have no other way of understanding what exactly developers are doing, either because they've never written software before or because they are too far removed from the level of the guys actually writing code to pay attention to what they're actually doing beyond just looking at a single number. It makes sense for managers who don't know what engineers are doing to seek out a different way to measure their performance and progress. The solution isn't do nothing, and it's not story points or any of that BS. But what are the alternatives? The only one I can think of is, don't put a manager in a position to manage individual developers if he/she is unable to understand what those individuals are even doing. Having an experienced senior engineer with the right skillset for management directly in charge of every team is amazingly valuable in my opinion. It's also vital that those higher up on the ladder are able to trust these manager-engineers, who are basically acting as an abstraction layer between the regular engineers and higher management. A lot of teams already have this in the form of a senior engineer or software architect, my team has one and it works out amazingly. We all feel that doing a good job will be recognized and rewarded, problems are easily identified and addressed, we get things done very quickly, everyone gets along, and there's not a lot of pressure. This is just my own single anecdotal experience and it has worked out very well but I'm aware it may not always be that way, so I'm interested in hearing others thoughts/experiences/solutions

Fire-Dragon-DoL 7y ago
Just to highlight this, this is what Toyota does with its engineers (not limited to software). Managers are engineers of towering technical competence.

iudqnolq 7y ago

I think a better version is "you get what you measure". Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n. (I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/ https://xkcd.com/810/)

plorkyeran 7y ago

If you reward a team for increasing story points completed, then you will very quickly find that every single story has the maximum possible estimate. In the longer term you'd get stories split into more smaller stories far more often than is reasonable. It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.

ratww 7y ago
I saw that happen in a team whose OKR was stupidly tied to story points. They went from 20 points per sprint to 70 overnight, by splitting stories as most as possible. It was hilarious when they reached their quarter OKR in less than a month.
- asplake 7y ago
  I have heard this described as “quantitative easing” for story points, and yes, hilarious.

jghn 7y ago

It wasn’t that long ago that OPS was not rewarded, and batting average was king. It took a lot of energy to change that tide and reward players for focusing on other metrics. My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.

ben509 7y ago

I think your counterexamples reveal why those metrics work: * In baseball, batting average and OPS require you to compete against other teams. * In finance, ROI is tested in the marketplace. * In sales, conversions require you to convince a skeptical customer. The software metrics are all things we have fairly direct control over. Maybe that suggests an amendment to Godhart's law: > When a measure that is controlled by the measuree becomes a target, it ceases to be a good measure.

mnowicki 7y ago
I was just writing a post that said the same exact thing with about 12 times as many words. Good thing I read your comment before I wasted another 20 minutes writing the other half of my response

chiefalchemist 7y ago

Maybe. But are you going to deduct for bugs? Or other flaws due to the metric not being "done right, and complete"?