4 ms·

This is such a compelling article on Goodharts law and interesting to see how Amazon conducts their Weekly Business Reviews. I'm wondering if there's a rebuttal

This is such a compelling article on Goodharts law and interesting to see how Amazon conducts their Weekly Business Reviews. I'm wondering if there's a rebuttal to this piece.

vaidhy 4y ago
Not a direct rebuttal.. but there are several things that need to come together for this to work. 1. The leadership team should trust each other's data. Over a period of time, that has not been true. 2. The group reviewing the data depends on everyone being there long enough to understand and get the feel. That is also hard to achieve and imho, is no longer true. 3. The metrics tend to ignore the cost to the workers. After all, everything looks like numbers and charts. For me, WBR is a mechanism, but Amazon's focus on just the metrics has hurt its workers, employees, sellers and partners a lot.
- andrewflnr 4y ago
  By "no longer", do you mean at Amazon particularly?
  - vaidhy 4y ago
    Yes.. I mean at Amazon. The WBR (when I was there) was held on Monday at the director level, Tuesday at the VP and Wednesday at the S-team. The following is purely my opinion: The constant churn of employees and directors at the lower levels meant the data is mostly massaged or explained away. The really good old-timers would focus on exceptions and single anomalies - much to the frustration of poor engineers and managers. But they intuitively knew that those small issues can snowball and catching them early helped mitigating the issue. But as the systems grew more complex and the number of teams expanded massively, the feel for the pulse became more mechanical and remote. New managers, directors and VPs did not come from a similar culture and the constant politics sidelined the more experienced employees who quit. Once that knowledge is lost, it takes multiple years to get it back and I do not think Amazon got that mojo anymore. It felt more mechanical and ritualistic, rather than genuinely trying to uncover issues (which is when I quit). It is Goodhart's law at the next level.
rocqua 4y ago
Looking at the 'damage' amazon has done, both to its workers, and to the credibility of their wares, you could say that amazon has suffered from goodhart's law to some extend on the grander scale. Specifically, they set a list of 'outcome metrics' that they wanted to optimize. Using these WBRs they ensures that the metrics by which their middle managers are measured remain effective for actually optimizing these outcome metrics. However, at some point these outcome metrics stopped being good targets. They have led amazon to exploit their workers, and have negative impact on outside businesses and the environment to the cost of massive political pressure. Moreover, these targets have led amazon to often sell counterfeit products, as well as incredibly low-quality budget alternatives. This is slowly causing people to distrust the amazon store for many things. In other words, whatever outcome metrics they set as their target, have seized being a good metric for how well amazon is doing. Or maybe the problems I identify are well considered trade-offs, and they are aware this is what their targets are causing...
- hgsgm 4y ago
  You not liking Amazon's goals doesn't make them bad for Amazon. Unless your concerns lead to >$500B loss, they are merely tradeoffs in the overall optimization.
  - taeric 4y ago
    The entire overbuild and overhire of the pandemic? Hard not to think of the hiring metrics that you know they were tracking in "efficient WBR" meetings the entire time. :D
itsdrewmiller 4y ago
What part of it do you want to see rebutted? I thought it was a great article too, but it seems like for the WBR process to work you need rapidly moving input and output metrics, which is more true of sales and marketing than engineering. Maybe you can apply the same approach to simple A/B tests, but then structurally you're incentivizing thinking small which puts you right back in the passengers seat of Goodhart's law.
snarf21 4y ago
> if you want to improve some process, you have to ignore the goal (target) This is the whole message of Goodhart's Law. Once you set a target, people will naturally aim for it. If you have to ignore the target or even not have one then you are saying Goodhart's Law is useful, which counters his narrative. The point of the measure should be to detect anomalies in the system, not drive targets or goals. If you measure widgets per week, you can look back at the end of each week and ask the questions: * Why did we produce 10% more widgets this week than our 6 week rolling average? * Why did we produce 5% less widgets this week than our 6 week rolling average? * Is there anything we can easily do to increase the amount of widgets we produce next week? None of these questions have a target. They are process oriented. We aren't saying what the rolling average must be, we are using it to detect variance. Or, we are investigating to see if we can produce positive variance. What we try might fail and the variance is negative but then we revert our process. Or it is positive and we have improved our process. Rinse and repeat
tqi 4y ago
I mentioned it in a different comment, but my biggest reservation is that the difference between "Distorting the system" and "Improving the system" is more often than not in the eye of the beholder, and most metrics will incentivize a mixed bag of "good" behavior and "distorting" behavior.

This is such a compelling article on Goodharts law and interesting to see how Amazon conducts their Weekly Business Reviews. I'm wondering if there's a rebuttal to this piece.

vaidhy 4y ago

Not a direct rebuttal.. but there are several things that need to come together for this to work. 1. The leadership team should trust each other's data. Over a period of time, that has not been true. 2. The group reviewing the data depends on everyone being there long enough to understand and get the feel. That is also hard to achieve and imho, is no longer true. 3. The metrics tend to ignore the cost to the workers. After all, everything looks like numbers and charts. For me, WBR is a mechanism, but Amazon's focus on just the metrics has hurt its workers, employees, sellers and partners a lot.

andrewflnr 4y ago
By "no longer", do you mean at Amazon particularly?
- vaidhy 4y ago
  Yes.. I mean at Amazon. The WBR (when I was there) was held on Monday at the director level, Tuesday at the VP and Wednesday at the S-team. The following is purely my opinion: The constant churn of employees and directors at the lower levels meant the data is mostly massaged or explained away. The really good old-timers would focus on exceptions and single anomalies - much to the frustration of poor engineers and managers. But they intuitively knew that those small issues can snowball and catching them early helped mitigating the issue. But as the systems grew more complex and the number of teams expanded massively, the feel for the pulse became more mechanical and remote. New managers, directors and VPs did not come from a similar culture and the constant politics sidelined the more experienced employees who quit. Once that knowledge is lost, it takes multiple years to get it back and I do not think Amazon got that mojo anymore. It felt more mechanical and ritualistic, rather than genuinely trying to uncover issues (which is when I quit). It is Goodhart's law at the next level.

rocqua 4y ago

Looking at the 'damage' amazon has done, both to its workers, and to the credibility of their wares, you could say that amazon has suffered from goodhart's law to some extend on the grander scale. Specifically, they set a list of 'outcome metrics' that they wanted to optimize. Using these WBRs they ensures that the metrics by which their middle managers are measured remain effective for actually optimizing these outcome metrics. However, at some point these outcome metrics stopped being good targets. They have led amazon to exploit their workers, and have negative impact on outside businesses and the environment to the cost of massive political pressure. Moreover, these targets have led amazon to often sell counterfeit products, as well as incredibly low-quality budget alternatives. This is slowly causing people to distrust the amazon store for many things. In other words, whatever outcome metrics they set as their target, have seized being a good metric for how well amazon is doing. Or maybe the problems I identify are well considered trade-offs, and they are aware this is what their targets are causing...

hgsgm 4y ago
You not liking Amazon's goals doesn't make them bad for Amazon. Unless your concerns lead to >$500B loss, they are merely tradeoffs in the overall optimization.
- taeric 4y ago
  The entire overbuild and overhire of the pandemic? Hard not to think of the hiring metrics that you know they were tracking in "efficient WBR" meetings the entire time. :D

itsdrewmiller 4y ago

What part of it do you want to see rebutted? I thought it was a great article too, but it seems like for the WBR process to work you need rapidly moving input and output metrics, which is more true of sales and marketing than engineering. Maybe you can apply the same approach to simple A/B tests, but then structurally you're incentivizing thinking small which puts you right back in the passengers seat of Goodhart's law.

snarf21 4y ago

> if you want to improve some process, you have to ignore the goal (target) This is the whole message of Goodhart's Law. Once you set a target, people will naturally aim for it. If you have to ignore the target or even not have one then you are saying Goodhart's Law is useful, which counters his narrative. The point of the measure should be to detect anomalies in the system, not drive targets or goals. If you measure widgets per week, you can look back at the end of each week and ask the questions: * Why did we produce 10% more widgets this week than our 6 week rolling average? * Why did we produce 5% less widgets this week than our 6 week rolling average? * Is there anything we can easily do to increase the amount of widgets we produce next week? None of these questions have a target. They are process oriented. We aren't saying what the rolling average must be, we are using it to detect variance. Or, we are investigating to see if we can produce positive variance. What we try might fail and the variance is negative but then we revert our process. Or it is positive and we have improved our process. Rinse and repeat

tqi 4y ago

I mentioned it in a different comment, but my biggest reservation is that the difference between "Distorting the system" and "Improving the system" is more often than not in the eye of the beholder, and most metrics will incentivize a mixed bag of "good" behavior and "distorting" behavior.