6 ms·

KPIs are not a shit system, they are actually important to running any nontrivial organization. The issue isn't whether KPIs are good or bad but what you do wi

KPIs are not a shit system, they are actually important to running any nontrivial organization. The issue isn't whether KPIs are good or bad but what you do with them. KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc. For example, a number of users resigning from our services might be a KPI. KPIs are an important tool to understand what is going on and whether we are going in the desired direction or not and to communicate this to your teams. KPIs are a tool for telling your organization what you think is important, to set the incentives in the right direction. KPIs will not help you if you have no effing idea how to set incentives -- they are just a tool in case you know what you are doing. Because they are just one step from being a target for your teams, you need to be very careful to communicate what you mean exactly when you set KPIs. Now, if you make number of users resigning from your services a KPI, you immediately need to set some additional guarantees that this is not overused -- for example by making it difficult for your clients to resign. This might be another KPI (user satisfaction with regards to how easy it was for them to resign).

Clewza313 5y ago
Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.
- lmilcin 5y ago
  Overuse law. If you overuse a law too much it stops being meaningful. Organizations have hard time improving without measuring their performance and communicating incentives. There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success. The best what you can do is compromise, it is unavoidable. So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.
  - Tainnor 5y ago
    The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.
    - dtech 5y ago
      You can't have a (semi-objective) target without a metric.
      - Tainnor 5y ago
        What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
        andrey_utkin 5y ago
        IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
- shkkmo 5y ago
  Which would indicate that the issue is not using KPIs to understand how your business is performing, but providing incentives based on KPIs.
  - Ericson2314 5y ago
    This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too. I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
    - Ericson2314 5y ago
      Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time. Mondragon please get in the tech biz.
      - andrey_utkin 5y ago
        Could you please show some real-world examples of this practice?
        Ericson2314 5y ago
        I am familiar with tech co-ops, but they are small, and so not good evidence.
      - nsonha 5y ago
        Didn't Donal Trump fool a lot of people in a democracy? And how does this have anything to do with metrics?
        patrick451 5y ago
        No, he didn't fool us.
- croes 5y ago
  Rather Marilyn Strathern's rephrasing. The original quote is "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
alxlaz 5y ago
> KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc. Right, but my post, the post I was responding to, the post the post I was responding to was responding to, and the article in the link, was specifically about a situation where KPIs are tied to actual people. I'm not sure what you're arguing for or against.
fungiblecog 5y ago
Measuring things is important, but turning them into a target is a disaster. The term KPI is generally used when the measure is a target.
mbrodersen 5y ago
I have seen many examples of KPI’s not working as intended. Do you have any real world examples of KPI’s that actually worked in practice (and not just in the make-believe heads of managers)?

Clewza313 5y ago

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.

lmilcin 5y ago
Overuse law. If you overuse a law too much it stops being meaningful. Organizations have hard time improving without measuring their performance and communicating incentives. There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success. The best what you can do is compromise, it is unavoidable. So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.
- Tainnor 5y ago
  The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.
  - dtech 5y ago
    You can't have a (semi-objective) target without a metric.
    - Tainnor 5y ago
      What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
      - andrey_utkin 5y ago
        IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
shkkmo 5y ago
Which would indicate that the issue is not using KPIs to understand how your business is performing, but providing incentives based on KPIs.
- Ericson2314 5y ago
  This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too. I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
  - Ericson2314 5y ago
    Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time. Mondragon please get in the tech biz.
    - andrey_utkin 5y ago
      Could you please show some real-world examples of this practice?
      - Ericson2314 5y ago
        I am familiar with tech co-ops, but they are small, and so not good evidence.
    - nsonha 5y ago
      Didn't Donal Trump fool a lot of people in a democracy? And how does this have anything to do with metrics?
      - patrick451 5y ago
        No, he didn't fool us.
croes 5y ago
Rather Marilyn Strathern's rephrasing. The original quote is "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

alxlaz 5y ago

> KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc. Right, but my post, the post I was responding to, the post the post I was responding to was responding to, and the article in the link, was specifically about a situation where KPIs are tied to actual people. I'm not sure what you're arguing for or against.

fungiblecog 5y ago

Measuring things is important, but turning them into a target is a disaster. The term KPI is generally used when the measure is a target.

mbrodersen 5y ago

I have seen many examples of KPI’s not working as intended. Do you have any real world examples of KPI’s that actually worked in practice (and not just in the make-believe heads of managers)?