8 ms·

Goodhart's Law

178 points by rfreytag 5y ago

rfreytag 5y ago
Earlier posts here (134 comments): https://news.ycombinator.com/item?id=17320640 https://news.ycombinator.com/item?id=17320640 and here (58 comments): https://news.ycombinator.com/item?id=23762526 https://news.ycombinator.com/item?id=23762526 Also NPR's Planet Money (audio) also covered this interviewing Goodhart himself: https://www.npr.org/sections/money/2018/11/19/669395064/episode-877-the-laws-of-the-office?t=1631892970800 https://www.npr.org/sections/money/2018/11/19/669395064/epis...
pikwip 5y ago
Here's a interesting paper I found that attempts to categorize the mechanisms by which Goodhart's Law operates in the real world. The variants are separated into Causal and Non-causal mechanisms. https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585
- derbOac 5y ago
  Thanks for posting that. I was going to say -- it's interesting to think about the reasons why Goodhart's Law might hold if it does. I've always assumed the problem is that the metric is always influenced by other, nontarget variables that become more causally important when the metric becomes a proxy target. So, for example, "gaming the metric" becomes important (in a percent variance sense) after the metric becomes a target. I think the paper's adversarial scenario is closest to this maybe. They discuss some other factors that seem more relevant to individual cases at any moment in time than an explanation for why a metric's utility might decline over time. In that sense the paper seems to be more about Goodheart-like phenomena in general. It would be interesting to demonstrate Goodheart's law conclusively with real data in some domains.
- davidmanheim 5y ago
  Thanks for highlighting our paper!
vagab0nd 5y ago
Here's another interesting one that's somewhat related: https://en.wikipedia.org/wiki/Decline_effect https://en.wikipedia.org/wiki/Decline_effect
- rfreytag 5y ago
  Could be...yes. "Decline effect" could also be mostly due to the https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Goodhart's Law starts out as effective and the social system it purports to measure adapts (some might say 'distorts'), till the measure not longer serves its original purpose.
brightball 5y ago
One of the most important lessons to promote.
baron_harkonnen 5y ago
I once mentioned Goodhart's Law to a data scientist at a company and they immediately rejected it based on the unironic assertion that "that would mean that KPIs shouldn't be the sole measure of our performance that that doesn't make sense!" My experience in the field has been that an astounding number of products have been destroyed and users harmed by failing to heed Goodhart's Law.
- some_furry 5y ago
  I wonder if measuring managers' understanding of Goodhart's Law would result in better management. /s
- dhosek 5y ago
  Are you sure it was unironic? Because I sure can't imagine anyone saying that unironically.
  - marcosdumay 5y ago
    I've heard plenty of "what else do I have to work with?", that has about the same meaning.
    - rhizome 5y ago
      The politician's syllogism comes to mind: "Something must be done, this is something, so we must do this." https://en.wikipedia.org/wiki/Politician%27s_syllogism https://en.wikipedia.org/wiki/Politician%27s_syllogism
    - gipp 5y ago
      That at least acknowledges that it's an unsatisfactory situation, OP's conversation didn't even have that level of awareness.
  - dboreham 5y ago
    Presumably someone, somewhere, thinks KPIs are a good idea. Edit: but that person is unlikely to be subject to KPIs themselves.
paulpauper 5y ago
This is why efforts at raising school test scores have not improved actual achievement
- dnautics 5y ago
  This is why efforts at alleviating poverty have not improved actual poverty...
  - umvi 5y ago
    Poverty is a special case though because it's relative. You could be a millionaire with a yacht on earth but be below the poverty line if the middle and upper classes live in space stations or on other planets with higher standards of living. An impoverished person in the US is rich compared to an impoverished person in India or Africa.
    - dnautics 5y ago
      That doesn't make my statement any less true.
  - Too 5y ago
    Source? According to the book Factfulnes by Hans Rosling, poverty has decreased significantly the past decades. https://www.amazon.com/Factfulness-Reasons-World-Things-Better/dp/1250107814 https://www.amazon.com/Factfulness-Reasons-World-Things-Bett...
- wyager 5y ago
  And why sending everyone to college hasn’t achieved anything good.
colechristensen 5y ago
The only systems that escape goodhearts or similar laws are those that have either A) people who genuinely care about quality and have the judgment for it or B) systems where the mechanics of how the control variable effects the system are well understood. (i.e. no black boxes)
- leephillips 5y ago
  I think the only way to apply a metric while avoiding the consequences of the Law is to keep the metric secret. As soon as the subjects are aware of the metric, they will try to optimize it, rather than the real performance that the metric is supposed to indicate. There is a close connection with security, say screening at airports. If you fail to keep your screening criteria secret, the terrorists will simply ensure that they do not match the criteria. One way to thwart this is through randomness. There is a long public debate between Sam Harris and Bruce Schneier where the latter tried in vain to explain this to the former, who insisted that it was a waste of resources to search little old ladies. If one of your metrics is “don’t search little old ladies”, the terrorists will discover this through time by observation. The next bomb will be carried by a little old lady.
_greim_ 5y ago
What's the takeaway? How can statistics inform action, if such action invalidates those statistics?
- cryptica 5y ago
  The take away is that governments and large institutions which impact large numbers of people should never attempt to set quantifiable targets and should never attempt to meet quantifiable targets.
- Jtsummers 5y ago
  That we can't stop thinking. It becomes too easy to stop using our own judgement when we have numbers to back up our decisions. We can point to he analysis and say, "Look, we've improved <metric>!". That itself becomes the justification for the actions taken, regardless of their actual sensibility. This is an area where discipline remains essential, and maintaining discipline is a constant battle.
- castlecrasher2 5y ago
  That statistics should inform action or measure of efficacy, but not drive it. A dev manager running on only numbers will inevitably get empty, meaningless values such as tickets resolved or lines of code written, while the polar opposite manager will run on intuition alone. I imagine most would agree neither type is generally effective and a balance should be struck, and Goodhart's Law means you should be aware of what's important, pay attention to it, but do not make it your sole focus. And for God's sake, don't make a public dashboard for it.
- ItsMonkk 5y ago
  You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible. Not for everyone, anyway. Using hiring as an example, but this should work for nearly all metrics driven workloads. Hire a small subset of your staff using a real judgment system, hire most of your staff using metrics, then take a sample of those hired using metrics and take a real look at them compared against those hired using real judgments. If they are reasonably close, the KPIs are working. If they are alien to each-other, you need to either stop using KPIs or alter them significantly to fit with what the more effective people are doing.
cs702 5y ago
The same phenomenon has many different names. From the OP: > See also: > Campbell's law – "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures" https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law > Cobra effect – when incentives designed to solve a problem end up rewarding people for making it worse https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect > Gaming the system https://en.wikipedia.org/wiki/Gaming_the_system https://en.wikipedia.org/wiki/Gaming_the_system > Lucas critique – it is naive to try to predict the effects of a change in economic policy entirely on the basis of relationships observed in historical data https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique > McNamara fallacy – involves making a decision based solely on quantitative observations (or metrics) and ignoring all others https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy > Overfitting https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting > Reflexivity (social theory) https://en.wikipedia.org/wiki/Reflexivity_(social_theory) https://en.wikipedia.org/wiki/Reflexivity_(social_theory) > Reification (fallacy) https://en.wikipedia.org/wiki/Reification_(fallacy) https://en.wikipedia.org/wiki/Reification_(fallacy) > San Francisco Declaration on Research Assessment – 2012 manifesto against using the journal impact factor to assess a scientist's work https://en.wikipedia.org/wiki/San_Francisco_Declaration_on_Research_Assessment https://en.wikipedia.org/wiki/San_Francisco_Declaration_on_R... > Volkswagen emissions scandal – 2010s diesel emissions scandal involving Volkswagen https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal Source: https://en.wikipedia.org/wiki/Goodhart%27s_law#See_also https://en.wikipedia.org/wiki/Goodhart%27s_law#See_also
azhenley 5y ago
I wrote a blog post a few months ago about Goodhart's Law in my life, titled "Gamification, life, and the pursuit of a gold badge". https://web.eecs.utk.edu/~azh/blog/gamification.html https://web.eecs.utk.edu/~azh/blog/gamification.html The Tyranny of Metrics is a good book that covers real-world cases of metrics gone wrong.
rpdillon 5y ago
A closely-related effect I often cite is the McNamara fallacy[0], which is essentially about the tendency to focus on aspects of a system that are easily measurable, often at the expense of aspects that are not. I see it as one of the weaknesses of the data-driven decision-making movement, since many interpret "data" to mean "numbers". I think this fallacy can partly explain why Goodhart's Law holds: it's the non-measurable (or difficult-to-measure) aspects that suffer most when a metric becomes a target, since measurable aspects could be (and often are) integrated into the target metric. [0]: https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy
- ghaff 5y ago
  It's a genuinely hard problem because the quantifiable output metrics are easy to measure and an individual often does have some level of direct control over them. So we convince ourselves that they're a reasonable proxy for something we care about but aren't sure how to measure and that we have control systems in place, whether management or individual responsibility, that largely prevent e.g. quality being thrown away in pursuit of quantity. And we're often not entirely wrong if we do pick reasonable proxies and have reasonable control systems in place. Because throwing up our hands and saying metrics are useless is usually not the answer either.
  - TeMPOraL 5y ago
    A flavor of this problem is what could be called "diffusion of responsibility", for a lack of better term. An individual who defined the measure and then optimizes for it will quickly figure out when their measure stops being a good proxy. But in organizations there usually isn't a single person who both understands what the measures are proxying, and has the power to remove a metric once it is used up, or get people to stop overfitting it.
    - Jtsummers 5y ago
      Another thing I've observed is when you have two measures that operate at different time scales. Both may even be valid measures, but the one measured (and responded to) more often has a stronger impact, and can negatively impact the less frequently measured metric when there's a conflict between them. A particular instance for this has been (to keep it simple) quantity (speed) and quality in production environments (factories and the like). Daily throughput measures paired with less frequent quality measures. The desire is to keep throughput high, and quality ends up suffering as a result. By integrating quality measures into the process you make the two measures compete on more equal footing, forcing a balance. At least one factory I worked in (well, adjacent to, I was in the software portion not the assembly line) massively reduced their quality problems by integrating quality checks between each station. This contrasted with the prior years where throughput, being measured and reacted to daily, drove them to make things so fast that they had piles of rework at the end. Integrating the quality measures between stations slowed them down, but their rework numbers turned into a rounding error (over a decade ago so I've forgotten the exact numbers, but they went from having items needing rework nearly every day to maybe one or two a month). As a result their real (deliverable to customers) production increased and their cost per unit dropped.
paulorlando 5y ago
Good that this topic gets some attention. I see Goodhart's Law again and again in metrics. I wrote about this a while back, including why we use the misleading name. https://unintendedconsequenc.es/new-morality-of-attainment-goodharts-law/ https://unintendedconsequenc.es/new-morality-of-attainment-g...
dang 5y ago
Past related threads. In this case a few of the 1-or-2 comment threads have particularly good posts: Goodhart's Law - https://news.ycombinator.com/item?id=26839177 https://news.ycombinator.com/item?id=26839177 - April 2021 (2 comments) Goodhart’s Law Rules the Modern World. Here Are Nine Examples - https://news.ycombinator.com/item?id=26604130 https://news.ycombinator.com/item?id=26604130 - March 2021 (3 comments) Goodhart's Law and how systems are shaped by the metrics you chase - https://news.ycombinator.com/item?id=23762526 https://news.ycombinator.com/item?id=23762526 - July 2020 (58 comments) When Goodharting Is Optimal - https://news.ycombinator.com/item?id=22054359 https://news.ycombinator.com/item?id=22054359 - Jan 2020 (3 comments) Goodhart’s Law: Are Academic Metrics Being Gamed? - https://news.ycombinator.com/item?id=21065507 https://news.ycombinator.com/item?id=21065507 - Sept 2019 (27 comments) Goodhart’s Law: Are Academic Metrics Being Gamed? - https://news.ycombinator.com/item?id=20076485 https://news.ycombinator.com/item?id=20076485 - June 2019 (2 comments) When targets and metrics are bad for business - https://news.ycombinator.com/item?id=19135694 https://news.ycombinator.com/item?id=19135694 - Feb 2019 (6 comments) Goodhart's Law: When a measure becomes a target, it ceases to be a good measure - https://news.ycombinator.com/item?id=17320640 https://news.ycombinator.com/item?id=17320640 - June 2018 (134 comments) Goodhart's Law - https://news.ycombinator.com/item?id=10075780 https://news.ycombinator.com/item?id=10075780 - Aug 2015 (1 comment) Goodhart's law - https://news.ycombinator.com/item?id=1368745 https://news.ycombinator.com/item?id=1368745 - May 2010 (1 comment)
bedhead 5y ago
I do investment stuff for a living and this was one of the single-most important things I ever learned.
starnger 5y ago
I could not help but relate it to Heisenberg uncertainty principle.
- thekhatribharat 5y ago
  me too :)
MattGaiser 5y ago
The ignoring of this law in software development is referred to as Scrum.
nicodds 5y ago
It is like quantum mechanics: the measurement process produces a perturbation of the physical system
deleted 5y ago
[deleted]
Beldin 5y ago
Inspired by a scandalous fraud case in scientific publishing, i wrote a paper applying Goodhart's Law to scientific publishing ("A Much-needed Security Perspective on Publication Metrics", published at the Security Protocols Workshop 2017). That was a really fun paper to write! Basically, how can you systematically start cheating at publishing - and how could you catch that? The most fun was challenging the audience - security researchers all - to think even more outside the box than usual for them. I'm still (slowly) forging ahead on ideas spawned by this paper. Bringing the ideas of catching crooks to reality was not as straightforward as hoped. Then again, when has any project ever gone as planned?
gftsantana 5y ago
My first job was at the anti-fraud department at a telecom company in the early 00s. Our job was basically to determine whether a new landline or mobile contract was fraudulent or not. Some requests would be flagged by an automated piece of software that was basically a black box to most employees, myself included. We would basically look at the documents sent by the clients and, sometimes, ask a few questions via phone. I was very young at the time, but I remember basically deriving Goodhart's law after a few months in the job. I don't remember clearly most of the things that led me to that conclusion, but I do remember the most extreme: at some point, management started requiring us to block clearly non-fraudulent phones because the directors decided to increase the blocked installations target. It would include even old contracts by good paying customers that happened to be flagged. I remember trying to talk to people about this, but the idea that trying to reach a target by any means necessary is usually not a good idea was incomprehensible to most people. Years later, I realized that others knew exactly what was going on; they just didn't care, and I was naive for not seeing that. It was a few years later when I learned about perverse incentives, Goodhart's law, the cobra effect, etc., and it allowed me to have more productive conversations with people about targets and incentives.
lisper 5y ago
It is possible to turn this effect to your advantage. I wrote a spam filter that takes advantage of signals that would be easy for spammers to spoof (like the list-id header), but no one spoofs them because no one but me uses this approach to spam filtering. So, ironically, if more people used my spam filter, it would probably stop working as well as it does now.
cratermoon 5y ago
When I explain the concept to people I usually call it the Goodhart-Strathern principle, to recognize the generalization she contributed and acknowledge the author of the most-commonly quoted form of the law: "When a measure becomes a target, it ceases to be a good measure"
equality_1138 5y ago
Reminded me also of the Taguchi loss function. To paraphrase, quality is optimized when you manage the variation of a measurement in relation to its target value, instead of emphasizing the acceptable limits. It was seen as an improvement over the "goalpost" approach. And while you can say it really applies in manufacturing context, I think it should be considered more broadly for goal-oriented measurement and engineering in general. https://en.wikipedia.org/wiki/Taguchi_loss_function https://en.wikipedia.org/wiki/Taguchi_loss_function
ausbah 5y ago
sounds like SOTA ml research

178 points by rfreytag 5y ago

Earlier posts here (134 comments): https://news.ycombinator.com/item?id=17320640 https://news.ycombinator.com/item?id=17320640 and here (58 comments): https://news.ycombinator.com/item?id=23762526 https://news.ycombinator.com/item?id=23762526 Also NPR's Planet Money (audio) also covered this interviewing Goodhart himself: https://www.npr.org/sections/money/2018/11/19/669395064/episode-877-the-laws-of-the-office?t=1631892970800 https://www.npr.org/sections/money/2018/11/19/669395064/epis...

pikwip 5y ago

Here's a interesting paper I found that attempts to categorize the mechanisms by which Goodhart's Law operates in the real world. The variants are separated into Causal and Non-causal mechanisms. https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585

derbOac 5y ago
Thanks for posting that. I was going to say -- it's interesting to think about the reasons why Goodhart's Law might hold if it does. I've always assumed the problem is that the metric is always influenced by other, nontarget variables that become more causally important when the metric becomes a proxy target. So, for example, "gaming the metric" becomes important (in a percent variance sense) after the metric becomes a target. I think the paper's adversarial scenario is closest to this maybe. They discuss some other factors that seem more relevant to individual cases at any moment in time than an explanation for why a metric's utility might decline over time. In that sense the paper seems to be more about Goodheart-like phenomena in general. It would be interesting to demonstrate Goodheart's law conclusively with real data in some domains.
davidmanheim 5y ago
Thanks for highlighting our paper!

vagab0nd 5y ago

Here's another interesting one that's somewhat related: https://en.wikipedia.org/wiki/Decline_effect https://en.wikipedia.org/wiki/Decline_effect

rfreytag 5y ago
Could be...yes. "Decline effect" could also be mostly due to the https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Goodhart's Law starts out as effective and the social system it purports to measure adapts (some might say 'distorts'), till the measure not longer serves its original purpose.

brightball 5y ago

One of the most important lessons to promote.

baron_harkonnen 5y ago

I once mentioned Goodhart's Law to a data scientist at a company and they immediately rejected it based on the unironic assertion that "that would mean that KPIs shouldn't be the sole measure of our performance that that doesn't make sense!" My experience in the field has been that an astounding number of products have been destroyed and users harmed by failing to heed Goodhart's Law.

some_furry 5y ago
I wonder if measuring managers' understanding of Goodhart's Law would result in better management. /s
dhosek 5y ago
Are you sure it was unironic? Because I sure can't imagine anyone saying that unironically.
- marcosdumay 5y ago
  I've heard plenty of "what else do I have to work with?", that has about the same meaning.
  - rhizome 5y ago
    The politician's syllogism comes to mind: "Something must be done, this is something, so we must do this." https://en.wikipedia.org/wiki/Politician%27s_syllogism https://en.wikipedia.org/wiki/Politician%27s_syllogism
  - gipp 5y ago
    That at least acknowledges that it's an unsatisfactory situation, OP's conversation didn't even have that level of awareness.
- dboreham 5y ago
  Presumably someone, somewhere, thinks KPIs are a good idea. Edit: but that person is unlikely to be subject to KPIs themselves.

paulpauper 5y ago

This is why efforts at raising school test scores have not improved actual achievement

dnautics 5y ago
This is why efforts at alleviating poverty have not improved actual poverty...
- umvi 5y ago
  Poverty is a special case though because it's relative. You could be a millionaire with a yacht on earth but be below the poverty line if the middle and upper classes live in space stations or on other planets with higher standards of living. An impoverished person in the US is rich compared to an impoverished person in India or Africa.
  - dnautics 5y ago
    That doesn't make my statement any less true.
- Too 5y ago
  Source? According to the book Factfulnes by Hans Rosling, poverty has decreased significantly the past decades. https://www.amazon.com/Factfulness-Reasons-World-Things-Better/dp/1250107814 https://www.amazon.com/Factfulness-Reasons-World-Things-Bett...
wyager 5y ago
And why sending everyone to college hasn’t achieved anything good.

colechristensen 5y ago

The only systems that escape goodhearts or similar laws are those that have either A) people who genuinely care about quality and have the judgment for it or B) systems where the mechanics of how the control variable effects the system are well understood. (i.e. no black boxes)

leephillips 5y ago
I think the only way to apply a metric while avoiding the consequences of the Law is to keep the metric secret. As soon as the subjects are aware of the metric, they will try to optimize it, rather than the real performance that the metric is supposed to indicate. There is a close connection with security, say screening at airports. If you fail to keep your screening criteria secret, the terrorists will simply ensure that they do not match the criteria. One way to thwart this is through randomness. There is a long public debate between Sam Harris and Bruce Schneier where the latter tried in vain to explain this to the former, who insisted that it was a waste of resources to search little old ladies. If one of your metrics is “don’t search little old ladies”, the terrorists will discover this through time by observation. The next bomb will be carried by a little old lady.

_greim_ 5y ago

What's the takeaway? How can statistics inform action, if such action invalidates those statistics?

cryptica 5y ago
The take away is that governments and large institutions which impact large numbers of people should never attempt to set quantifiable targets and should never attempt to meet quantifiable targets.
Jtsummers 5y ago
That we can't stop thinking. It becomes too easy to stop using our own judgement when we have numbers to back up our decisions. We can point to he analysis and say, "Look, we've improved <metric>!". That itself becomes the justification for the actions taken, regardless of their actual sensibility. This is an area where discipline remains essential, and maintaining discipline is a constant battle.
castlecrasher2 5y ago
That statistics should inform action or measure of efficacy, but not drive it. A dev manager running on only numbers will inevitably get empty, meaningless values such as tickets resolved or lines of code written, while the polar opposite manager will run on intuition alone. I imagine most would agree neither type is generally effective and a balance should be struck, and Goodhart's Law means you should be aware of what's important, pay attention to it, but do not make it your sole focus. And for God's sake, don't make a public dashboard for it.
ItsMonkk 5y ago
You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible. Not for everyone, anyway. Using hiring as an example, but this should work for nearly all metrics driven workloads. Hire a small subset of your staff using a real judgment system, hire most of your staff using metrics, then take a sample of those hired using metrics and take a real look at them compared against those hired using real judgments. If they are reasonably close, the KPIs are working. If they are alien to each-other, you need to either stop using KPIs or alter them significantly to fit with what the more effective people are doing.

cs702 5y ago

The same phenomenon has many different names. From the OP: > See also: > Campbell's law – "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures" https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law > Cobra effect – when incentives designed to solve a problem end up rewarding people for making it worse https://en.wikipedia.org/wiki/Cobra_effect https://en.wikipedia.org/wiki/Cobra_effect > Gaming the system https://en.wikipedia.org/wiki/Gaming_the_system https://en.wikipedia.org/wiki/Gaming_the_system > Lucas critique – it is naive to try to predict the effects of a change in economic policy entirely on the basis of relationships observed in historical data https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique > McNamara fallacy – involves making a decision based solely on quantitative observations (or metrics) and ignoring all others https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy > Overfitting https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting > Reflexivity (social theory) https://en.wikipedia.org/wiki/Reflexivity_(social_theory) https://en.wikipedia.org/wiki/Reflexivity_(social_theory) > Reification (fallacy) https://en.wikipedia.org/wiki/Reification_(fallacy) https://en.wikipedia.org/wiki/Reification_(fallacy) > San Francisco Declaration on Research Assessment – 2012 manifesto against using the journal impact factor to assess a scientist's work https://en.wikipedia.org/wiki/San_Francisco_Declaration_on_Research_Assessment https://en.wikipedia.org/wiki/San_Francisco_Declaration_on_R... > Volkswagen emissions scandal – 2010s diesel emissions scandal involving Volkswagen https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal Source: https://en.wikipedia.org/wiki/Goodhart%27s_law#See_also https://en.wikipedia.org/wiki/Goodhart%27s_law#See_also

azhenley 5y ago

I wrote a blog post a few months ago about Goodhart's Law in my life, titled "Gamification, life, and the pursuit of a gold badge". https://web.eecs.utk.edu/~azh/blog/gamification.html https://web.eecs.utk.edu/~azh/blog/gamification.html The Tyranny of Metrics is a good book that covers real-world cases of metrics gone wrong.

rpdillon 5y ago

A closely-related effect I often cite is the McNamara fallacy[0], which is essentially about the tendency to focus on aspects of a system that are easily measurable, often at the expense of aspects that are not. I see it as one of the weaknesses of the data-driven decision-making movement, since many interpret "data" to mean "numbers". I think this fallacy can partly explain why Goodhart's Law holds: it's the non-measurable (or difficult-to-measure) aspects that suffer most when a metric becomes a target, since measurable aspects could be (and often are) integrated into the target metric. [0]: https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy

ghaff 5y ago
It's a genuinely hard problem because the quantifiable output metrics are easy to measure and an individual often does have some level of direct control over them. So we convince ourselves that they're a reasonable proxy for something we care about but aren't sure how to measure and that we have control systems in place, whether management or individual responsibility, that largely prevent e.g. quality being thrown away in pursuit of quantity. And we're often not entirely wrong if we do pick reasonable proxies and have reasonable control systems in place. Because throwing up our hands and saying metrics are useless is usually not the answer either.
- TeMPOraL 5y ago
  A flavor of this problem is what could be called "diffusion of responsibility", for a lack of better term. An individual who defined the measure and then optimizes for it will quickly figure out when their measure stops being a good proxy. But in organizations there usually isn't a single person who both understands what the measures are proxying, and has the power to remove a metric once it is used up, or get people to stop overfitting it.
  - Jtsummers 5y ago
    Another thing I've observed is when you have two measures that operate at different time scales. Both may even be valid measures, but the one measured (and responded to) more often has a stronger impact, and can negatively impact the less frequently measured metric when there's a conflict between them. A particular instance for this has been (to keep it simple) quantity (speed) and quality in production environments (factories and the like). Daily throughput measures paired with less frequent quality measures. The desire is to keep throughput high, and quality ends up suffering as a result. By integrating quality measures into the process you make the two measures compete on more equal footing, forcing a balance. At least one factory I worked in (well, adjacent to, I was in the software portion not the assembly line) massively reduced their quality problems by integrating quality checks between each station. This contrasted with the prior years where throughput, being measured and reacted to daily, drove them to make things so fast that they had piles of rework at the end. Integrating the quality measures between stations slowed them down, but their rework numbers turned into a rounding error (over a decade ago so I've forgotten the exact numbers, but they went from having items needing rework nearly every day to maybe one or two a month). As a result their real (deliverable to customers) production increased and their cost per unit dropped.

paulorlando 5y ago

Good that this topic gets some attention. I see Goodhart's Law again and again in metrics. I wrote about this a while back, including why we use the misleading name. https://unintendedconsequenc.es/new-morality-of-attainment-goodharts-law/ https://unintendedconsequenc.es/new-morality-of-attainment-g...

dang 5y ago

Past related threads. In this case a few of the 1-or-2 comment threads have particularly good posts: Goodhart's Law - https://news.ycombinator.com/item?id=26839177 https://news.ycombinator.com/item?id=26839177 - April 2021 (2 comments) Goodhart’s Law Rules the Modern World. Here Are Nine Examples - https://news.ycombinator.com/item?id=26604130 https://news.ycombinator.com/item?id=26604130 - March 2021 (3 comments) Goodhart's Law and how systems are shaped by the metrics you chase - https://news.ycombinator.com/item?id=23762526 https://news.ycombinator.com/item?id=23762526 - July 2020 (58 comments) When Goodharting Is Optimal - https://news.ycombinator.com/item?id=22054359 https://news.ycombinator.com/item?id=22054359 - Jan 2020 (3 comments) Goodhart’s Law: Are Academic Metrics Being Gamed? - https://news.ycombinator.com/item?id=21065507 https://news.ycombinator.com/item?id=21065507 - Sept 2019 (27 comments) Goodhart’s Law: Are Academic Metrics Being Gamed? - https://news.ycombinator.com/item?id=20076485 https://news.ycombinator.com/item?id=20076485 - June 2019 (2 comments) When targets and metrics are bad for business - https://news.ycombinator.com/item?id=19135694 https://news.ycombinator.com/item?id=19135694 - Feb 2019 (6 comments) Goodhart's Law: When a measure becomes a target, it ceases to be a good measure - https://news.ycombinator.com/item?id=17320640 https://news.ycombinator.com/item?id=17320640 - June 2018 (134 comments) Goodhart's Law - https://news.ycombinator.com/item?id=10075780 https://news.ycombinator.com/item?id=10075780 - Aug 2015 (1 comment) Goodhart's law - https://news.ycombinator.com/item?id=1368745 https://news.ycombinator.com/item?id=1368745 - May 2010 (1 comment)

bedhead 5y ago

I do investment stuff for a living and this was one of the single-most important things I ever learned.

starnger 5y ago

I could not help but relate it to Heisenberg uncertainty principle.

thekhatribharat 5y ago
me too :)

MattGaiser 5y ago

The ignoring of this law in software development is referred to as Scrum.

nicodds 5y ago

It is like quantum mechanics: the measurement process produces a perturbation of the physical system

deleted 5y ago

[deleted]

Beldin 5y ago

Inspired by a scandalous fraud case in scientific publishing, i wrote a paper applying Goodhart's Law to scientific publishing ("A Much-needed Security Perspective on Publication Metrics", published at the Security Protocols Workshop 2017). That was a really fun paper to write! Basically, how can you systematically start cheating at publishing - and how could you catch that? The most fun was challenging the audience - security researchers all - to think even more outside the box than usual for them. I'm still (slowly) forging ahead on ideas spawned by this paper. Bringing the ideas of catching crooks to reality was not as straightforward as hoped. Then again, when has any project ever gone as planned?

gftsantana 5y ago

My first job was at the anti-fraud department at a telecom company in the early 00s. Our job was basically to determine whether a new landline or mobile contract was fraudulent or not. Some requests would be flagged by an automated piece of software that was basically a black box to most employees, myself included. We would basically look at the documents sent by the clients and, sometimes, ask a few questions via phone. I was very young at the time, but I remember basically deriving Goodhart's law after a few months in the job. I don't remember clearly most of the things that led me to that conclusion, but I do remember the most extreme: at some point, management started requiring us to block clearly non-fraudulent phones because the directors decided to increase the blocked installations target. It would include even old contracts by good paying customers that happened to be flagged. I remember trying to talk to people about this, but the idea that trying to reach a target by any means necessary is usually not a good idea was incomprehensible to most people. Years later, I realized that others knew exactly what was going on; they just didn't care, and I was naive for not seeing that. It was a few years later when I learned about perverse incentives, Goodhart's law, the cobra effect, etc., and it allowed me to have more productive conversations with people about targets and incentives.

lisper 5y ago

It is possible to turn this effect to your advantage. I wrote a spam filter that takes advantage of signals that would be easy for spammers to spoof (like the list-id header), but no one spoofs them because no one but me uses this approach to spam filtering. So, ironically, if more people used my spam filter, it would probably stop working as well as it does now.

cratermoon 5y ago

When I explain the concept to people I usually call it the Goodhart-Strathern principle, to recognize the generalization she contributed and acknowledge the author of the most-commonly quoted form of the law: "When a measure becomes a target, it ceases to be a good measure"

equality_1138 5y ago

Reminded me also of the Taguchi loss function. To paraphrase, quality is optimized when you manage the variation of a measurement in relation to its target value, instead of emphasizing the acceptable limits. It was seen as an improvement over the "goalpost" approach. And while you can say it really applies in manufacturing context, I think it should be considered more broadly for goal-oriented measurement and engineering in general. https://en.wikipedia.org/wiki/Taguchi_loss_function https://en.wikipedia.org/wiki/Taguchi_loss_function

ausbah 5y ago

sounds like SOTA ml research