4 ms·

You can't have a (semi-objective) target without a metric.

by dtech 5y ago

Tainnor 5y ago
What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
- andrey_utkin 5y ago
  IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
- ghaff 5y ago
  As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc. Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level. The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.

You can't have a (semi-objective) target without a metric.

by dtech 5y ago

You can't have a (semi-objective) target without a metric.

What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in. You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed. Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy. I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.

andrey_utkin 5y ago
IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers). If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed. But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
ghaff 5y ago
As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc. Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level. The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.