15 ms·

Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

229 points by chrisshroba 8y ago

mathattack 8y ago
I’ve seen this a lot. Even people who intellectually know that correlation doesn’t equal causality make this mistake all the time.
- smallnamespace 8y ago
  It's not even that correlation doesn't equal causality, it's that the correlation may also completely reverse.
mlthoughts2018 8y ago
So we should minimize the fraction of measures which have become targets then?
- smallnamespace 8y ago
  You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them. That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something that ceases to improve their life quality.
  - maxxxxx 8y ago
    Exactly. Don't reduce everything down to one factor. In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors. Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things. Focusing on only one thing while neglecting others doesn't work.
    - mlthoughts2018 8y ago
      But then your "one factor" just becomes a combination of various other factors. I don't think the point of Goodhart's Law is related to "over-optimizing for just one thing," rather it's a statement about whether any metric (even a broad, diverse metric that takes into account multiple different things) can retain informativeness once it is set up as an optimization target.
      - maxxxxx 8y ago
        I think you could set up an optimization target for a company that says "be profitable and grow for the next 30 years". You would see quickly that this target is a complex beast with a lot of different factors at play that need to be balanced against each other. I don't think Goodhart's law would apply here.
  - davidmanheim 8y ago
    That's often helpful, but not sufficient in general. See the dynamics discussed in my article here; https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585
  - mlthoughts2018 8y ago
    That just sounds like coordinate gradient descent on a multi-term objective function. Just same old optimization but with some metaparameters or component-wise conditioning. In fact, I think to argue against optimization, you’d basically have to boil it down to existing philosophical arguments against utilitarianism. Either way, I’m skeptical if this conclusion is very connected to Goodhart’s Law, because it assumes some type of normative axiom that would have to precede your interpretation of Goodhart’s Law. My main interest was to pointout that it creates a sort of impossibility theorem situation if you accept Goodhart’s Law (I don’t personally accept it as a general rule, only something that applies to some types of measures and not others). The impossibility criteria would be something like: - If one finds a good measure, one should optimize it. - If a measure is an optimization target, it is not a good measure. - For a given entity, there exists a set of variables governing how well that entity can meet a goal (e.g. measures can possibly contain relevant information). Basically you could only choose 2. - If you pick the first two, it means there are no general variables governing the outcone you want; they are constantly changing and fundamentally unpredictable, so you might get away with optimizing something briefly, but its informativeness immediately disappears. This type of thinking is common in efficient market theory, for example, and could be connected to a fundamental belief about non-determinism or non-stationarity. - If you choose the first and third, it seems like standard utilitarianism. - If you choose the latter two, it seems like fatalism or religion — there are quantities that govern achieving desired outcomes, but they cannot be treated as objective optimization criteria and can only be analyzed subjectively or fatalistically.
- squidbidness 8y ago
  I love the irony in this.
nikanj 8y ago
"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
- tomjen3 8y ago
  That is only true if you do that with no regard for the way it was achieved.
  - supercheetah 8y ago
    That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.
  - acdha 8y ago
    That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
  - setr 8y ago
    the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.
- personjerry 8y ago
  So then you add “size of customer base” to your metrics!
  - scj 8y ago
    Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted). In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.
pwagland 8y ago
Indeed this is the whole idea behind OKR. The Objectives are _supposed_ to be grand, audacious, and only achievable under extreme stretching. Furthermore, they are _supposed_ to not be used as a KPI for performance reviews.
mitko 8y ago
I've seen similar topics come up on HN front page every once in a while. Here are some of my thoughts on the topic, through the prism of metrics for a company. http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-paradox http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-para... TLDR of my post: The vanity metric paradox - every metric, sufficiently optimized becomes a vanity metric, as it stops being the make-or-break qualifier for a company.
kolbe 8y ago
I wish more people knew and understood this concept. It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.
- velp 8y ago
  Can you explain how the recession was caused by "government's choice of credit worthiness metrics", instead of rollbacks of regulation?
  - kolbe 8y ago
    I'm not sure why you think these things are mutually exclusive. And as far as Goodhart's Law is concerned in this case, they're intertwined concepts. The government's choice to allow securities rated by S&P/Moody's to be treated as gospel allowed/encouraged banks, insurance companies, GSE's and pension funds to take far more risk than was systemically reasonable.
RelaxSelf 8y ago
yep, seen several idiots whose work habits are destructive be promoted for gaming stats seen stats make people discourage/delay income/work to avoid anomalies/spikes in stats because if there's a spike it makes it look like you are slacking the next month, or makes a manager think you should be able to constantly hit the spike
stcredzero 8y ago
During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.
- AcerbicZero 8y ago
  Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.
  - stcredzero 8y ago
    That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example. There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.
    - cryoshon 8y ago
      in principle, you can verify every downed aircraft and destroyed tank with a photograph or a video. it's hard to move the wrecks of each. there are worse metrics.
      - stcredzero 8y ago
        But still, such figures as reported were inflated by large margins (something on the order of 25 to 100 percent) on both the Allied and Axis sides.
    - sonnyblarney 8y ago
      I'm sorry but this is a serious (and arguably offensive) falsehood. A 'free fire' zone is an area wherein there is no requirement for soldiers to coordinate heavy arms fire with other units via HQ etc.. This is a very reasonable policy in jungle fights where units can come under ambush at any time, i.e. "We're in an ambush and we're about to die, can you ring up HQ to see if we can shoot back?" But the notion that US forces would just designate an area where everyone including civilians are to be killed is an outright and offensive lie. Though there are some situations in which Army Units did actively and knowingly fire on citizens, these are known and well documented tragedies of soldiers acting with fury in the moment agains villagers who were supporting insurgent forces - this was not and never was any kind of US policy, and the perpetrators faced Court Martial. The Vietnam war is so completely misinterpreted in pop culture; even to this day, it's disturbing.
inanutshellus 8y ago
And as soon as a measure becomes known, it becomes the target. IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure." "I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks." "What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."
- spodek 8y ago
  > And as soon as a measure becomes known, it becomes the target. Soon after, 90% of the target is still an 'A', which becomes the new target. On a related note of unintended consequences, the Cobra Effect: https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect
- niftich 8y ago
  Gamification increases as understanding improves. Information is exploited.
  - stcredzero 8y ago
    [Targeting Intensifies]
- coredog64 8y ago
  Before the VW diesel cheating scandal, it was alleged that US manufacturers were taking advantage of the EPA's test cycle procedure to "tune" ECM values when the dome light was active. More than that, CAFE standards have become the ceiling for fuel economy rather than the envisioned floor. Manufacturers game the system by classifying certain cars as SUVs, claiming extra credit for the ability to run E85, or practically giving away efficient but undesirable vehicle configurations in order to just meet the target and avoid the gas guzzler tax.
- megablast 8y ago
  Sweet. The measure is profit then.
Angostura 8y ago
Serious question - it may no longer be a good measure, but does that necessarily mean it is a poor target? Assuming targets are necessary, do they not have to based on done measure?
biastoact 8y ago
Without dates and metrics you can’t measure progress. Like mile markers along the road (back before GPS / Smartphones). But without continuous, high trust, collaborative conversation you can’t actually make progress. That’s the gasoline powering you down the road. The goal should be Disneyland (shouldn’t it always be Disneyland?), not the next mile marker.
notEvenOnce 8y ago
(cough! cough! "scrum") (cough! cough! "agile") (cough! cough! "daily stand-ups")
- jingleheimer 8y ago
  I always felt that agile processes were a means to give non-answers to performance measurement in order to allay the consequences of that measurement and allow developers to get on with the business of making software. How big is the task; 10 story points How long will that take; depends on our velocity Can it be done by this date; we're not allowed to plan that far ahead. You can put it in the 'icebox'. We might get to it eventually.
  - pmwhite 8y ago
    Yes, definitely true. If (1) everyone is highly effective in getting tasks done, and (2) working on tasks that are reasonably well prioritized by both technical risks and business value, why exactly do you need to know more? There are legitimate reasons that a business may need a date. But making sure people are working harder than hard by beating them over the head with a date coughed out by a gantt chart is not one of the legitimate reasons.
- ozim 8y ago
  Kind of we started optimizing for points and started to be smart about it. Because that what people above wanted. What happened we started communicating to PO more often, like: you want more points burned take this story out, close this and create an improvement. Of course that was not childish silly sneaking and turned out we have as a team a lot better partnership with business. We deliver what is needed and can be finished. I understand not everyone has luck to end up in such environment but it confirms there is no silver bullet. Only having right people is real silver bullet.
logicallee 8y ago
Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?
- mannykannot 8y ago
  Yes; the real problem is when a proxy is substituted for what is really wanted - for example, when lines of code written is taken for a measure of how much usefulness has been created, or the percentage graduated is taken as a measure of the effectiveness of education. Proxies are not intrinsically bad, either, but they have to be difficult to achieve without also achieving what you really want - when this is not the case, making the proxy measurement a target motivates people to seek workarounds. Life expectancy might be a proxy, but it is a pretty good one.
- bennofs 8y ago
  You could get a long life expectancy but with increasing pain as you get older. It would be a long live but not healthy. See also https://en.m.wikipedia.org/wiki/Compression_of_morbidity https://en.m.wikipedia.org/wiki/Compression_of_morbidity
- scarmig 8y ago
  That's easy, though. If you're going for life expectancy, then there's plenty of things you can do. 1) Exclude deaths that'd hurt your life expectancy. For instance, by under counting infant mortality. E.g. does a child who dies during birth count against your life expectancy? Arguments can go both ways, but if your hospital's funding depends on having good life expectancy numbers in your community... well, there you go. IIRC this is one factor that helps explain Cuba's surprisingly high life expectancy numbers, though it doesn't account for most of it. 2) Overtreat end of life issues. Instead of focusing on quality of life and general health throughout the lifespan, put all your funding into heroic interventions that let 90 year olds live on artificial respiration for months. This is what the United States does. 3) Systematically deceive. Have a significant homeless population? Rural population that's deeply unhealthy? Make sure you put a ton of resources into measuring outcomes for your urban population with relatively good health access, and the peasants you miss are just a sign of not getting enough grant money from the WHO. Governments that put resources into getting measurements of the most marginal populations are penalized, while those that deprioritize studies of those marginal populations would have better numbers for their marketing brochures. Even though we'd expect the former to have actual better outcomes than the latter!
davidmanheim 8y ago
This is probably where I should mention my two longer articles that discuss this in depth; https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-measurement-is-hard/ https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-... https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspecified-goals/ https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspec... As well as my more recent paper categorizing the different ways it occurs; https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585
- Symmetry 8y ago
  That was a much better take on Goodhart's Law than the Wikipedia article. It's easy to give up and say "Measurement is impossible!" but really we have to balance different factors and measurements are more dangerous at some times than at others.
  - rsync 8y ago
    Lots of content at ribbonfarm is very, very good ... highly recommended ... https://www.ribbonfarm.com/2010/11/05/ancient-rivers-of-money/ https://www.ribbonfarm.com/2010/11/05/ancient-rivers-of-mone... https://www.ribbonfarm.com/2009/03/02/fools-and-their-money-metaphors/ https://www.ribbonfarm.com/2009/03/02/fools-and-their-money-... The Return of the Barbarian Can Hydras eat unknown unknowns for lunch The Premium Mediocre Life of Maya Millennial
- waltherg 8y ago
  Since you’re here, what does the following property stand for in your second paper? s ∈ AifM(s)/gec
  - defen 8y ago
    Not OP but that should probably render as: s ∈ A if M(s) >= c So: s (system state) is a member of A (permissible region) if M(s) (proxy measure of s since we can't directly observe it) is greater than or equal to c (threshold)
  - deleted 8y ago
yeeeeeeeeee 8y ago
Measurements are chords in a concert.
njarboe 8y ago
The breaking of this law when one uses profit as the measure of a successful company or business is why I think that a well set up market system actually works. People and companies can set up making a profit as their target and it is still a good measure of success, because it is actually what you want. The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not. A hard thing to get right but forcing people to inform you of the ingredients of food for sale, is an example of a good regulation I think.
- retbull 8y ago
  Unless the highest profit thing is to poison a town full of people. Which has happened when profit is what you measure as success.
  - njarboe 8y ago
    Correct. "The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not." Start of the second paragraph.
- jetrink 8y ago
  The implication behind Goodhart's Law is that people are good at gaming performance measures. You can game profit measures too[1], but it's difficult and there are strong incentives not to. Another counter-example that doesn't involve fraud is Amazon, which was unprofitable for a long time. 1. https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal
  - dfee 8y ago
    “Profits” in your context being imagined not real. The gaming was done on the accounting side (revenue recognition) of course.
- nkoren 8y ago
  Even disregarding the potential for negative externalities etc., profit alone can still be a terrible KPI. I worked for a large engineering consultancy that was very profit motivated. They figured out a brilliant way to maximise profit: get rid of business development! Only pay for work when clients are paying you to do it. This was GREAT for quarterly profit margins, at the expense of the project pipeline. As the pipeline dried up, they figured out a new way to maximize profit: get rid of engineers! The customers are big, stupid organisations, and it'll take them a long time to notice. You can totally maximise quarterly profit margins until they do. And that's how you can simultaneously be highly profitable and in a corporate death spiral. Moral of the story: reality is a fundamentally complex place. You need multiple KPIs to get a handle on it -- and ideally those KPIs should be mutually contradictory enough to force you to engage with that complexity head-on, rather than boiling things down to game-able metrics which ultimately will defeat themselves because they are such a poor proxy for reality.
dfee 8y ago
Another article on the front page decries Harvard’s holistic approach to admission - with arguments that admission should be based on a single measure of academic performance (like standardized test scores, GPA, etc.).
- lostcolony 8y ago
  Yes, but holistic approaches have -also- become a target. Think of the over-achieving high schoolers doing as many extra-curriculars as possible just for their college application.
- stevenwoo 8y ago
  The most interesting approach I have seen as an alternative to current ones is the one that suggests emphasizing the student body of the school - leading parents to self integrate their K-12 schools on racial and socioeconomic factors. This would be unassailable under the current legal attacks used to fight against affirmative action. https://ylpr.yale.edu/shifting-scope-how-taking-school-demographics-account-college-admissions-could-reduce-k-12 https://ylpr.yale.edu/shifting-scope-how-taking-school-demog...
gojomo 8y ago
And thus for performance measures, "security through obscurity" can make sense.
triviatise 8y ago
We hold people accountable for actions instead of outcomes, since outcomes (even profit) can be gamed at the overall detriment of the organization. We talk about the outcome we want, look at the levers that will get the outcome, and then agree to the actions that we are going to take. If the actions dont result in the forecasted outcome then the underlying assumptions are wrong. The assumptions are wrong if 1) people arent doing the tasks properly 2) the relationship between the task and the outcome are different than what we thought. If the issue is 1, we provide training. If the issue is 2, we use the revised assumption for future forecasts. We dont use the word target anymore, we use the word forecast.
fallingfrog 8y ago
That's why you know your company is in trouble when shareholder becomes the customer and the stock price becomes the product..
dqpb 8y ago
This is one reason why the grading system in schools is absurd and incompetent.
joker3 8y ago
People respond more or less rationally to their incentives. Who knew?
stcredzero 8y ago
The same principle applies to attributes and qualities which mark successful entrepreneurs. The same principle applies to attributes and qualities which mark good developers. The same principle applies to attributes and qualities which mark good employees. Good people. Good politicians. Activists. Good pundits. Intellectuals. Top tier school degrees. Hacker News karma. Shouldn't we all be looking out for the "targeting" and be trying to find the next measure?
tlarkworthy 8y ago
This is also demonstrated quite well in machine learning with the test set. The target is test error, but you don't let algorithms optimize that directly and instead they optimize against training error. Goodhart's law is a manifestation of over-fitting in social systems.
em3rgent0rdr 8y ago
Reminds me of all those computer architecture papers that optimize for IPC to the detriment of overall performance. Or the CPU frequency wars which led to ever deeper pipelines which were completely obliterated with branch mispredicts. Or the current trend of increasing core count without actually being able to do any useful work on them...

229 points by chrisshroba 8y ago

I’ve seen this a lot. Even people who intellectually know that correlation doesn’t equal causality make this mistake all the time.

smallnamespace 8y ago
It's not even that correlation doesn't equal causality, it's that the correlation may also completely reverse.

mlthoughts2018 8y ago

So we should minimize the fraction of measures which have become targets then?

smallnamespace 8y ago
You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them. That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something that ceases to improve their life quality.
- maxxxxx 8y ago
  Exactly. Don't reduce everything down to one factor. In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors. Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things. Focusing on only one thing while neglecting others doesn't work.
  - mlthoughts2018 8y ago
    But then your "one factor" just becomes a combination of various other factors. I don't think the point of Goodhart's Law is related to "over-optimizing for just one thing," rather it's a statement about whether any metric (even a broad, diverse metric that takes into account multiple different things) can retain informativeness once it is set up as an optimization target.
    - maxxxxx 8y ago
      I think you could set up an optimization target for a company that says "be profitable and grow for the next 30 years". You would see quickly that this target is a complex beast with a lot of different factors at play that need to be balanced against each other. I don't think Goodhart's law would apply here.
- davidmanheim 8y ago
  That's often helpful, but not sufficient in general. See the dynamics discussed in my article here; https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585
- mlthoughts2018 8y ago
  That just sounds like coordinate gradient descent on a multi-term objective function. Just same old optimization but with some metaparameters or component-wise conditioning. In fact, I think to argue against optimization, you’d basically have to boil it down to existing philosophical arguments against utilitarianism. Either way, I’m skeptical if this conclusion is very connected to Goodhart’s Law, because it assumes some type of normative axiom that would have to precede your interpretation of Goodhart’s Law. My main interest was to pointout that it creates a sort of impossibility theorem situation if you accept Goodhart’s Law (I don’t personally accept it as a general rule, only something that applies to some types of measures and not others). The impossibility criteria would be something like: - If one finds a good measure, one should optimize it. - If a measure is an optimization target, it is not a good measure. - For a given entity, there exists a set of variables governing how well that entity can meet a goal (e.g. measures can possibly contain relevant information). Basically you could only choose 2. - If you pick the first two, it means there are no general variables governing the outcone you want; they are constantly changing and fundamentally unpredictable, so you might get away with optimizing something briefly, but its informativeness immediately disappears. This type of thinking is common in efficient market theory, for example, and could be connected to a fundamental belief about non-determinism or non-stationarity. - If you choose the first and third, it seems like standard utilitarianism. - If you choose the latter two, it seems like fatalism or religion — there are quantities that govern achieving desired outcomes, but they cannot be treated as objective optimization criteria and can only be analyzed subjectively or fatalistically.
squidbidness 8y ago
I love the irony in this.

nikanj 8y ago

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

tomjen3 8y ago
That is only true if you do that with no regard for the way it was achieved.
- supercheetah 8y ago
  That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.
- acdha 8y ago
  That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
- setr 8y ago
  the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.
personjerry 8y ago
So then you add “size of customer base” to your metrics!
- scj 8y ago
  Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted). In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.

pwagland 8y ago

Indeed this is the whole idea behind OKR. The Objectives are _supposed_ to be grand, audacious, and only achievable under extreme stretching. Furthermore, they are _supposed_ to not be used as a KPI for performance reviews.

mitko 8y ago

I've seen similar topics come up on HN front page every once in a while. Here are some of my thoughts on the topic, through the prism of metrics for a company. http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-paradox http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-para... TLDR of my post: The vanity metric paradox - every metric, sufficiently optimized becomes a vanity metric, as it stops being the make-or-break qualifier for a company.

kolbe 8y ago

I wish more people knew and understood this concept. It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.

velp 8y ago
Can you explain how the recession was caused by "government's choice of credit worthiness metrics", instead of rollbacks of regulation?
- kolbe 8y ago
  I'm not sure why you think these things are mutually exclusive. And as far as Goodhart's Law is concerned in this case, they're intertwined concepts. The government's choice to allow securities rated by S&P/Moody's to be treated as gospel allowed/encouraged banks, insurance companies, GSE's and pension funds to take far more risk than was systemically reasonable.

RelaxSelf 8y ago

yep, seen several idiots whose work habits are destructive be promoted for gaming stats seen stats make people discourage/delay income/work to avoid anomalies/spikes in stats because if there's a spike it makes it look like you are slacking the next month, or makes a manager think you should be able to constantly hit the spike

stcredzero 8y ago

During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.

AcerbicZero 8y ago
Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.
- stcredzero 8y ago
  That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example. There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.
  - cryoshon 8y ago
    in principle, you can verify every downed aircraft and destroyed tank with a photograph or a video. it's hard to move the wrecks of each. there are worse metrics.
    - stcredzero 8y ago
      But still, such figures as reported were inflated by large margins (something on the order of 25 to 100 percent) on both the Allied and Axis sides.
  - sonnyblarney 8y ago
    I'm sorry but this is a serious (and arguably offensive) falsehood. A 'free fire' zone is an area wherein there is no requirement for soldiers to coordinate heavy arms fire with other units via HQ etc.. This is a very reasonable policy in jungle fights where units can come under ambush at any time, i.e. "We're in an ambush and we're about to die, can you ring up HQ to see if we can shoot back?" But the notion that US forces would just designate an area where everyone including civilians are to be killed is an outright and offensive lie. Though there are some situations in which Army Units did actively and knowingly fire on citizens, these are known and well documented tragedies of soldiers acting with fury in the moment agains villagers who were supporting insurgent forces - this was not and never was any kind of US policy, and the perpetrators faced Court Martial. The Vietnam war is so completely misinterpreted in pop culture; even to this day, it's disturbing.

inanutshellus 8y ago

And as soon as a measure becomes known, it becomes the target. IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure." "I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks." "What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."

spodek 8y ago
> And as soon as a measure becomes known, it becomes the target. Soon after, 90% of the target is still an 'A', which becomes the new target. On a related note of unintended consequences, the Cobra Effect: https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect
niftich 8y ago
Gamification increases as understanding improves. Information is exploited.
- stcredzero 8y ago
  [Targeting Intensifies]
coredog64 8y ago
Before the VW diesel cheating scandal, it was alleged that US manufacturers were taking advantage of the EPA's test cycle procedure to "tune" ECM values when the dome light was active. More than that, CAFE standards have become the ceiling for fuel economy rather than the envisioned floor. Manufacturers game the system by classifying certain cars as SUVs, claiming extra credit for the ability to run E85, or practically giving away efficient but undesirable vehicle configurations in order to just meet the target and avoid the gas guzzler tax.
megablast 8y ago
Sweet. The measure is profit then.

Angostura 8y ago

Serious question - it may no longer be a good measure, but does that necessarily mean it is a poor target? Assuming targets are necessary, do they not have to based on done measure?

biastoact 8y ago

Without dates and metrics you can’t measure progress. Like mile markers along the road (back before GPS / Smartphones). But without continuous, high trust, collaborative conversation you can’t actually make progress. That’s the gasoline powering you down the road. The goal should be Disneyland (shouldn’t it always be Disneyland?), not the next mile marker.

notEvenOnce 8y ago

(cough! cough! "scrum") (cough! cough! "agile") (cough! cough! "daily stand-ups")

jingleheimer 8y ago
I always felt that agile processes were a means to give non-answers to performance measurement in order to allay the consequences of that measurement and allow developers to get on with the business of making software. How big is the task; 10 story points How long will that take; depends on our velocity Can it be done by this date; we're not allowed to plan that far ahead. You can put it in the 'icebox'. We might get to it eventually.
- pmwhite 8y ago
  Yes, definitely true. If (1) everyone is highly effective in getting tasks done, and (2) working on tasks that are reasonably well prioritized by both technical risks and business value, why exactly do you need to know more? There are legitimate reasons that a business may need a date. But making sure people are working harder than hard by beating them over the head with a date coughed out by a gantt chart is not one of the legitimate reasons.
ozim 8y ago
Kind of we started optimizing for points and started to be smart about it. Because that what people above wanted. What happened we started communicating to PO more often, like: you want more points burned take this story out, close this and create an improvement. Of course that was not childish silly sneaking and turned out we have as a team a lot better partnership with business. We deliver what is needed and can be finished. I understand not everyone has luck to end up in such environment but it confirms there is no silver bullet. Only having right people is real silver bullet.

logicallee 8y ago

Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?

mannykannot 8y ago
Yes; the real problem is when a proxy is substituted for what is really wanted - for example, when lines of code written is taken for a measure of how much usefulness has been created, or the percentage graduated is taken as a measure of the effectiveness of education. Proxies are not intrinsically bad, either, but they have to be difficult to achieve without also achieving what you really want - when this is not the case, making the proxy measurement a target motivates people to seek workarounds. Life expectancy might be a proxy, but it is a pretty good one.
bennofs 8y ago
You could get a long life expectancy but with increasing pain as you get older. It would be a long live but not healthy. See also https://en.m.wikipedia.org/wiki/Compression_of_morbidity https://en.m.wikipedia.org/wiki/Compression_of_morbidity
scarmig 8y ago
That's easy, though. If you're going for life expectancy, then there's plenty of things you can do. 1) Exclude deaths that'd hurt your life expectancy. For instance, by under counting infant mortality. E.g. does a child who dies during birth count against your life expectancy? Arguments can go both ways, but if your hospital's funding depends on having good life expectancy numbers in your community... well, there you go. IIRC this is one factor that helps explain Cuba's surprisingly high life expectancy numbers, though it doesn't account for most of it. 2) Overtreat end of life issues. Instead of focusing on quality of life and general health throughout the lifespan, put all your funding into heroic interventions that let 90 year olds live on artificial respiration for months. This is what the United States does. 3) Systematically deceive. Have a significant homeless population? Rural population that's deeply unhealthy? Make sure you put a ton of resources into measuring outcomes for your urban population with relatively good health access, and the peasants you miss are just a sign of not getting enough grant money from the WHO. Governments that put resources into getting measurements of the most marginal populations are penalized, while those that deprioritize studies of those marginal populations would have better numbers for their marketing brochures. Even though we'd expect the former to have actual better outcomes than the latter!

davidmanheim 8y ago

This is probably where I should mention my two longer articles that discuss this in depth; https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-measurement-is-hard/ https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-... https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspecified-goals/ https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspec... As well as my more recent paper categorizing the different ways it occurs; https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585

Symmetry 8y ago
That was a much better take on Goodhart's Law than the Wikipedia article. It's easy to give up and say "Measurement is impossible!" but really we have to balance different factors and measurements are more dangerous at some times than at others.
- rsync 8y ago
  Lots of content at ribbonfarm is very, very good ... highly recommended ... https://www.ribbonfarm.com/2010/11/05/ancient-rivers-of-money/ https://www.ribbonfarm.com/2010/11/05/ancient-rivers-of-mone... https://www.ribbonfarm.com/2009/03/02/fools-and-their-money-metaphors/ https://www.ribbonfarm.com/2009/03/02/fools-and-their-money-... The Return of the Barbarian Can Hydras eat unknown unknowns for lunch The Premium Mediocre Life of Maya Millennial
waltherg 8y ago
Since you’re here, what does the following property stand for in your second paper? s ∈ AifM(s)/gec
- defen 8y ago
  Not OP but that should probably render as: s ∈ A if M(s) >= c So: s (system state) is a member of A (permissible region) if M(s) (proxy measure of s since we can't directly observe it) is greater than or equal to c (threshold)
- deleted 8y ago

yeeeeeeeeee 8y ago

Measurements are chords in a concert.

njarboe 8y ago

The breaking of this law when one uses profit as the measure of a successful company or business is why I think that a well set up market system actually works. People and companies can set up making a profit as their target and it is still a good measure of success, because it is actually what you want. The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not. A hard thing to get right but forcing people to inform you of the ingredients of food for sale, is an example of a good regulation I think.

retbull 8y ago
Unless the highest profit thing is to poison a town full of people. Which has happened when profit is what you measure as success.
- njarboe 8y ago
  Correct. "The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not." Start of the second paragraph.
jetrink 8y ago
The implication behind Goodhart's Law is that people are good at gaming performance measures. You can game profit measures too[1], but it's difficult and there are strong incentives not to. Another counter-example that doesn't involve fraud is Amazon, which was unprofitable for a long time. 1. https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal
- dfee 8y ago
  “Profits” in your context being imagined not real. The gaming was done on the accounting side (revenue recognition) of course.
nkoren 8y ago
Even disregarding the potential for negative externalities etc., profit alone can still be a terrible KPI. I worked for a large engineering consultancy that was very profit motivated. They figured out a brilliant way to maximise profit: get rid of business development! Only pay for work when clients are paying you to do it. This was GREAT for quarterly profit margins, at the expense of the project pipeline. As the pipeline dried up, they figured out a new way to maximize profit: get rid of engineers! The customers are big, stupid organisations, and it'll take them a long time to notice. You can totally maximise quarterly profit margins until they do. And that's how you can simultaneously be highly profitable and in a corporate death spiral. Moral of the story: reality is a fundamentally complex place. You need multiple KPIs to get a handle on it -- and ideally those KPIs should be mutually contradictory enough to force you to engage with that complexity head-on, rather than boiling things down to game-able metrics which ultimately will defeat themselves because they are such a poor proxy for reality.

dfee 8y ago

Another article on the front page decries Harvard’s holistic approach to admission - with arguments that admission should be based on a single measure of academic performance (like standardized test scores, GPA, etc.).

lostcolony 8y ago
Yes, but holistic approaches have -also- become a target. Think of the over-achieving high schoolers doing as many extra-curriculars as possible just for their college application.
stevenwoo 8y ago
The most interesting approach I have seen as an alternative to current ones is the one that suggests emphasizing the student body of the school - leading parents to self integrate their K-12 schools on racial and socioeconomic factors. This would be unassailable under the current legal attacks used to fight against affirmative action. https://ylpr.yale.edu/shifting-scope-how-taking-school-demographics-account-college-admissions-could-reduce-k-12 https://ylpr.yale.edu/shifting-scope-how-taking-school-demog...

gojomo 8y ago

And thus for performance measures, "security through obscurity" can make sense.

triviatise 8y ago

We hold people accountable for actions instead of outcomes, since outcomes (even profit) can be gamed at the overall detriment of the organization. We talk about the outcome we want, look at the levers that will get the outcome, and then agree to the actions that we are going to take. If the actions dont result in the forecasted outcome then the underlying assumptions are wrong. The assumptions are wrong if 1) people arent doing the tasks properly 2) the relationship between the task and the outcome are different than what we thought. If the issue is 1, we provide training. If the issue is 2, we use the revised assumption for future forecasts. We dont use the word target anymore, we use the word forecast.

fallingfrog 8y ago

That's why you know your company is in trouble when shareholder becomes the customer and the stock price becomes the product..

dqpb 8y ago

This is one reason why the grading system in schools is absurd and incompetent.

joker3 8y ago

People respond more or less rationally to their incentives. Who knew?

stcredzero 8y ago

The same principle applies to attributes and qualities which mark successful entrepreneurs. The same principle applies to attributes and qualities which mark good developers. The same principle applies to attributes and qualities which mark good employees. Good people. Good politicians. Activists. Good pundits. Intellectuals. Top tier school degrees. Hacker News karma. Shouldn't we all be looking out for the "targeting" and be trying to find the next measure?

tlarkworthy 8y ago

This is also demonstrated quite well in machine learning with the test set. The target is test error, but you don't let algorithms optimize that directly and instead they optimize against training error. Goodhart's law is a manifestation of over-fitting in social systems.

em3rgent0rdr 8y ago

Reminds me of all those computer architecture papers that optimize for IPC to the detriment of overall performance. Or the CPU frequency wars which led to ever deeper pipelines which were completely obliterated with branch mispredicts. Or the current trend of increasing core count without actually being able to do any useful work on them...