4 ms·

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're re

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.

lmilcin 5y ago
Overuse law. If you overuse a law too much it stops being meaningful. Organizations have hard time improving without measuring their performance and communicating incentives. There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success. The best what you can do is compromise, it is unavoidable. So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.
- Tainnor 5y ago
  The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.
  - dtech 5y ago
    You can't have a (semi-objective) target without a metric.
    - Tainnor 5y ago
      What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
      - andrey_utkin 5y ago
        IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
      - ghaff 5y ago
        As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc. Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level. The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.
  - dragonwriter 5y ago
    > The point they're trying to make is, I believe, that metrics shouldn't be targets. Metrics are either (1) targets or (2) things that you are trying to analyze in relation to the metrics that are targets or (3) a waste of the time you spent gathering them. The reason metrics are often bad when they become targets is the metrics arr usually not actual direct measures of goals, but things assumed to be convenient proxies, but when you push hard on optimizing them, they stop being good proxies because as well as being easier to measure than the real objective, once you pick any low-hanging productive fruit they are inevitably easier to improve in ways that don’t improve the real objective as much as the proxy (or at all, or which negatively impact it.)
shkkmo 5y ago
Which would indicate that the issue is not using KPIs to understand how your business is performing, but providing incentives based on KPIs.
- Ericson2314 5y ago
  This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too. I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
  - Ericson2314 5y ago
    Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time. Mondragon please get in the tech biz.
    - andrey_utkin 5y ago
      Could you please show some real-world examples of this practice?
      - Ericson2314 5y ago
        I am familiar with tech co-ops, but they are small, and so not good evidence.
    - nsonha 5y ago
      Didn't Donal Trump fool a lot of people in a democracy? And how does this have anything to do with metrics?
      - patrick451 5y ago
        No, he didn't fool us.
croes 5y ago
Rather Marilyn Strathern's rephrasing. The original quote is "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.

lmilcin 5y ago

Overuse law. If you overuse a law too much it stops being meaningful. Organizations have hard time improving without measuring their performance and communicating incentives. There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success. The best what you can do is compromise, it is unavoidable. So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.

Tainnor 5y ago
The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.
- dtech 5y ago
  You can't have a (semi-objective) target without a metric.
  - Tainnor 5y ago
    What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
    - andrey_utkin 5y ago
      IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
    - ghaff 5y ago
      As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc. Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level. The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.
- dragonwriter 5y ago
  > The point they're trying to make is, I believe, that metrics shouldn't be targets. Metrics are either (1) targets or (2) things that you are trying to analyze in relation to the metrics that are targets or (3) a waste of the time you spent gathering them. The reason metrics are often bad when they become targets is the metrics arr usually not actual direct measures of goals, but things assumed to be convenient proxies, but when you push hard on optimizing them, they stop being good proxies because as well as being easier to measure than the real objective, once you pick any low-hanging productive fruit they are inevitably easier to improve in ways that don’t improve the real objective as much as the proxy (or at all, or which negatively impact it.)

shkkmo 5y ago

Which would indicate that the issue is not using KPIs to understand how your business is performing, but providing incentives based on KPIs.

Ericson2314 5y ago
This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too. I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
- Ericson2314 5y ago
  Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time. Mondragon please get in the tech biz.
  - andrey_utkin 5y ago
    Could you please show some real-world examples of this practice?
    - Ericson2314 5y ago
      I am familiar with tech co-ops, but they are small, and so not good evidence.
  - nsonha 5y ago
    Didn't Donal Trump fool a lot of people in a democracy? And how does this have anything to do with metrics?
    - patrick451 5y ago
      No, he didn't fool us.

croes 5y ago

Rather Marilyn Strathern's rephrasing. The original quote is "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."