4 ms·

Policy formulators really need to understand Goodhart's law - https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law - I

by sriku 5y ago

Policy formulators really need to understand Goodhart's law - https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law - If your "productivity" is measured and rewarded based on lines of code, you know what'll happen. - If a QA engineer is incentivised based on number of detected bugs, you know what'll happen - a lot of trivial bugs gets reported. - If a dev is incentivised based on number of bugs per line of code, fewer bugs will get reported. ... and that's just w.r.t. software metrics. At the country level, we have the GDP to thank for if a govt targets its policies towards influencing the GDP number upwards to the detriment of what the people and economy really need in their context.

coliveira 5y ago
The whole capitalist society is based on how much money (some) people have!
travisjungroth 5y ago
After all this metric stuff, I’m much more in line with the idea of embracing the difficulty of assessment. What are the exact metrics I need to get promoted? There are none. I have heuristics for when it’s beneficial to reward an employee and increase their influence. Even I don’t fully understand them. Here’s some vague hints. Meet those, and you’ll be promoted. At least it’s honest.
- bluGill 5y ago
  It is also correct. Eventually you will have someone great who fails to meet one metric, but you need to promote them before they decide they aren't going to be rewarded and leave. Which is really what this is all about: ensuring people who you want to keep stick around. The pay you give someone is in part a reflection on how willing you are to leave. If my boss reduced my wage to minimum wage I wouldn't quit today, but you can be sure my top priority would be finding a new job. I know it is possible for me to feed my family on minimum wage, but I wouldn't like the lifestyle that means and I wouldn't like the snub.
Sparkyte 5y ago
The best model for a company is based off it's performance as a product. If your products are doing well then the team gets more funding. If not doing well less funding and cuts would have to be provisioned from that. Performance is messaured in five points. Sales, adoption, support, dependency and need. Sales, how much money am I making. Adoption, how many users are joining. Support, how many people current actively use the product. Dependency, how dependent are it's users on the product. Need, how much we really want this product to be successful. Some products purely exist on a need basis.
- saurik 5y ago
  But sometimes, the reason you are failing is because you are missing a key person or lacking funding, so it isn't quite as easy as that (and this is even said as someone who thinks most software companies have 10-100x too many employees ;P). Honestly, I think the real problem is blindly following "metrics" of any form: metrics are input to a decision making process, but can almost never be directly optimized without causing some kind of serious mistake (which is sad, as humans are often biased; the real solution is thereby to not build large companies, lol).
- specialist 5y ago
  Aligning incentives would be so easy. And yet profit sharing is so rare.
- yourapostasy 5y ago
  I've seen the Goodhart'ing of this product orientation lead to bizarre product segmentation detrimental to positioning the company within the actual market. People keep trying to come up with a way to "systematize management" into synthesizing leadership, and I keep maintaining there is no substitute for actual leadership, which is as much an art form as non-trivial debugging and coding (especially at scale and speed).
usgroup 5y ago
I think this is not correct because many of our biggest and most successful companies stack rank. If there is a substantial productivity advantage to not doing it then the next company with a million person workforce has some low hanging fruit to pick.
- deleted 5y ago
  [deleted]
- mattnewton 5y ago
  Like an evolutionary process, companies are only ever graded on the sum of their policies. There is almost never a company that offers almost everything the same but no stack ranking; it would not surprise me to see that the market for labor is “inefficient” here. I wonder if there are management studies on introducing stack ranking to a unit inside of a company?
  - Applejinx 5y ago
    If there are, they are not framed in societal terms but in Darwinist terms. In regular society, it's more profitable to be a criminal/mugger/what have you, viewed entirely in a practical sense. It only becomes unprofitable when (a) there's law to prevent you and (b) there's a general idea that preventing you from doing that is correct in some way. If anybody's experimented with this, and I wouldn't be surprised if Amazon had as it seems like they'd try anything to see if it worked, I would guess that at scale stack ranking produces slightly more efficiency and productivity than it costs in harm to the well-being of workers. Since Amazon doesn't care at all about the well-being of workers OR employers… they're like a sort of pyre to the cause of scale and profitability… then stack ranking would stay. If it didn't return some arbitrarily small (or large, as far as we know) benefit to the company at the cost of the well-being of the workers, it would be gone. Somebody's got to have measured this by now, and all these tech unicorns are smart enough to ditch it if it was actually impinging on their productivity. It's not, it's impinging on worker conditions, so win/win as far as the tech giants are concerned.
- acdha 5y ago
  It's hard finding counter-examples since no two companies are in the same business and situation, but consider Microsoft: stack-ranking coincided with their lost decade where they stagnated, ceded control of mobile & web to Apple & Google, etc. A new CEO changed many things so it's hard to say whether stack-ranking was more a cause or symptom of Ballmer's bad management but from the outside it certainly seems like a healthier company now that it's gone. That's also important to think about from the other perspective: stack-ranking seems like a frictional cost — Microsoft would have been profitable no matter what since anything short of discontinuing Windows/Office would have left a massive cash flow, but it's harder to see where pushing people out cost you until after the fact. Amazon has enjoyed a massive first mover advantage from the cloud and they're certainly executing well now but I would not be surprised if, say, strong competition lead to a point in a few years where they might wish that they hadn't lost staff due to their checkered reputation for being an enjoyable place to work.
conjectures 5y ago
Yup. The thing that's disturbing about this is that someone senior must understand Goodhart's law in the company. Which makes it more likely the side-effects are, in fact, intended.
- michaelt 5y ago
  Plenty of managers know about Goodhart's Law - they just also know about the power of incentives / KPIs / OKRs when properly managed. And they imagine they'll properly manage them. After all, you can't believe in perverse incentives without first believing in incentives! People in business largely believe in sales people getting bonuses for closing big sales, CEOs getting bonuses for raising the stock price, and workers getting bonuses for delivering projects on time and under budget.
- PedroBatista 5y ago
  At the end of the day “someone senior” is just protecting their own neck. These people got there for being “a team player”, ie. They play the game, in the office, in emails, in meetings, at parties and many in their own heads since school age. At the end of the day they couldn’t give a shit about Goodhart’s law and all the book they were made to read in order to keep playing the game.
  - Aeolun 5y ago
    The thing is, unless it comes all the way from the top, it’s not generally a good idea to disagree too vehemently with your boss. Especially if he thinks stack ranking is the shit.
someguyorother 5y ago
Perhaps a more general observation is that a measure that faces pressure by intelligent agents needs to be monitored and adjusted by equally intelligent agents to patch the exploits. So you can either use a simple measure that doesn't work, or a complex measure that does work, but at potentially great cost (evaluating all the terms, bureaucratic inertia by the controllers, etc.) Some metrics seem to provide more of a free lunch - be more robust - than others. It's not obvious which are robust, however.
- damagednoob 5y ago
  Sounds like this would just lead to an arms race between smart managers and smart employees.
- abandonliberty 5y ago
  This is everything in society, and many problems arise because those manipulating the measure have more to gain by challenging it, while the checks and balances only face loss.
christophilus 5y ago
“All my life, I’ve been in the top 5% of people who understand incentives, and all my life, I’ve underestimated it.” — Charlie Munger Quoting from memory. It’s true that almost every time I see a really strange group behavior, it’s the natural result of poorly conceived incentives. This is a problem that is likely to be with us forever.

by sriku 5y ago

coliveira 5y ago

The whole capitalist society is based on how much money (some) people have!

travisjungroth 5y ago

After all this metric stuff, I’m much more in line with the idea of embracing the difficulty of assessment. What are the exact metrics I need to get promoted? There are none. I have heuristics for when it’s beneficial to reward an employee and increase their influence. Even I don’t fully understand them. Here’s some vague hints. Meet those, and you’ll be promoted. At least it’s honest.

bluGill 5y ago
It is also correct. Eventually you will have someone great who fails to meet one metric, but you need to promote them before they decide they aren't going to be rewarded and leave. Which is really what this is all about: ensuring people who you want to keep stick around. The pay you give someone is in part a reflection on how willing you are to leave. If my boss reduced my wage to minimum wage I wouldn't quit today, but you can be sure my top priority would be finding a new job. I know it is possible for me to feed my family on minimum wage, but I wouldn't like the lifestyle that means and I wouldn't like the snub.

Sparkyte 5y ago

The best model for a company is based off it's performance as a product. If your products are doing well then the team gets more funding. If not doing well less funding and cuts would have to be provisioned from that. Performance is messaured in five points. Sales, adoption, support, dependency and need. Sales, how much money am I making. Adoption, how many users are joining. Support, how many people current actively use the product. Dependency, how dependent are it's users on the product. Need, how much we really want this product to be successful. Some products purely exist on a need basis.

saurik 5y ago
But sometimes, the reason you are failing is because you are missing a key person or lacking funding, so it isn't quite as easy as that (and this is even said as someone who thinks most software companies have 10-100x too many employees ;P). Honestly, I think the real problem is blindly following "metrics" of any form: metrics are input to a decision making process, but can almost never be directly optimized without causing some kind of serious mistake (which is sad, as humans are often biased; the real solution is thereby to not build large companies, lol).
specialist 5y ago
Aligning incentives would be so easy. And yet profit sharing is so rare.
yourapostasy 5y ago
I've seen the Goodhart'ing of this product orientation lead to bizarre product segmentation detrimental to positioning the company within the actual market. People keep trying to come up with a way to "systematize management" into synthesizing leadership, and I keep maintaining there is no substitute for actual leadership, which is as much an art form as non-trivial debugging and coding (especially at scale and speed).

usgroup 5y ago

I think this is not correct because many of our biggest and most successful companies stack rank. If there is a substantial productivity advantage to not doing it then the next company with a million person workforce has some low hanging fruit to pick.

deleted 5y ago
[deleted]
mattnewton 5y ago
Like an evolutionary process, companies are only ever graded on the sum of their policies. There is almost never a company that offers almost everything the same but no stack ranking; it would not surprise me to see that the market for labor is “inefficient” here. I wonder if there are management studies on introducing stack ranking to a unit inside of a company?
- Applejinx 5y ago
  If there are, they are not framed in societal terms but in Darwinist terms. In regular society, it's more profitable to be a criminal/mugger/what have you, viewed entirely in a practical sense. It only becomes unprofitable when (a) there's law to prevent you and (b) there's a general idea that preventing you from doing that is correct in some way. If anybody's experimented with this, and I wouldn't be surprised if Amazon had as it seems like they'd try anything to see if it worked, I would guess that at scale stack ranking produces slightly more efficiency and productivity than it costs in harm to the well-being of workers. Since Amazon doesn't care at all about the well-being of workers OR employers… they're like a sort of pyre to the cause of scale and profitability… then stack ranking would stay. If it didn't return some arbitrarily small (or large, as far as we know) benefit to the company at the cost of the well-being of the workers, it would be gone. Somebody's got to have measured this by now, and all these tech unicorns are smart enough to ditch it if it was actually impinging on their productivity. It's not, it's impinging on worker conditions, so win/win as far as the tech giants are concerned.
acdha 5y ago
It's hard finding counter-examples since no two companies are in the same business and situation, but consider Microsoft: stack-ranking coincided with their lost decade where they stagnated, ceded control of mobile & web to Apple & Google, etc. A new CEO changed many things so it's hard to say whether stack-ranking was more a cause or symptom of Ballmer's bad management but from the outside it certainly seems like a healthier company now that it's gone. That's also important to think about from the other perspective: stack-ranking seems like a frictional cost — Microsoft would have been profitable no matter what since anything short of discontinuing Windows/Office would have left a massive cash flow, but it's harder to see where pushing people out cost you until after the fact. Amazon has enjoyed a massive first mover advantage from the cloud and they're certainly executing well now but I would not be surprised if, say, strong competition lead to a point in a few years where they might wish that they hadn't lost staff due to their checkered reputation for being an enjoyable place to work.

conjectures 5y ago

Yup. The thing that's disturbing about this is that someone senior must understand Goodhart's law in the company. Which makes it more likely the side-effects are, in fact, intended.

michaelt 5y ago
Plenty of managers know about Goodhart's Law - they just also know about the power of incentives / KPIs / OKRs when properly managed. And they imagine they'll properly manage them. After all, you can't believe in perverse incentives without first believing in incentives! People in business largely believe in sales people getting bonuses for closing big sales, CEOs getting bonuses for raising the stock price, and workers getting bonuses for delivering projects on time and under budget.
PedroBatista 5y ago
At the end of the day “someone senior” is just protecting their own neck. These people got there for being “a team player”, ie. They play the game, in the office, in emails, in meetings, at parties and many in their own heads since school age. At the end of the day they couldn’t give a shit about Goodhart’s law and all the book they were made to read in order to keep playing the game.
- Aeolun 5y ago
  The thing is, unless it comes all the way from the top, it’s not generally a good idea to disagree too vehemently with your boss. Especially if he thinks stack ranking is the shit.

someguyorother 5y ago

Perhaps a more general observation is that a measure that faces pressure by intelligent agents needs to be monitored and adjusted by equally intelligent agents to patch the exploits. So you can either use a simple measure that doesn't work, or a complex measure that does work, but at potentially great cost (evaluating all the terms, bureaucratic inertia by the controllers, etc.) Some metrics seem to provide more of a free lunch - be more robust - than others. It's not obvious which are robust, however.

damagednoob 5y ago
Sounds like this would just lead to an arms race between smart managers and smart employees.
abandonliberty 5y ago
This is everything in society, and many problems arise because those manipulating the measure have more to gain by challenging it, while the checks and balances only face loss.

christophilus 5y ago

“All my life, I’ve been in the top 5% of people who understand incentives, and all my life, I’ve underestimated it.” — Charlie Munger Quoting from memory. It’s true that almost every time I see a really strange group behavior, it’s the natural result of poorly conceived incentives. This is a problem that is likely to be with us forever.