4 ms·

What's the takeaway? How can statistics inform action, if such action invalidates those statistics?

cryptica 5y ago
The take away is that governments and large institutions which impact large numbers of people should never attempt to set quantifiable targets and should never attempt to meet quantifiable targets.
Jtsummers 5y ago
That we can't stop thinking. It becomes too easy to stop using our own judgement when we have numbers to back up our decisions. We can point to he analysis and say, "Look, we've improved <metric>!". That itself becomes the justification for the actions taken, regardless of their actual sensibility. This is an area where discipline remains essential, and maintaining discipline is a constant battle.
castlecrasher2 5y ago
That statistics should inform action or measure of efficacy, but not drive it. A dev manager running on only numbers will inevitably get empty, meaningless values such as tickets resolved or lines of code written, while the polar opposite manager will run on intuition alone. I imagine most would agree neither type is generally effective and a balance should be struck, and Goodhart's Law means you should be aware of what's important, pay attention to it, but do not make it your sole focus. And for God's sake, don't make a public dashboard for it.
ItsMonkk 5y ago
You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible. Not for everyone, anyway. Using hiring as an example, but this should work for nearly all metrics driven workloads. Hire a small subset of your staff using a real judgment system, hire most of your staff using metrics, then take a sample of those hired using metrics and take a real look at them compared against those hired using real judgments. If they are reasonably close, the KPIs are working. If they are alien to each-other, you need to either stop using KPIs or alter them significantly to fit with what the more effective people are doing.
- rhizome 5y ago
  >You need to measure the KPIs. What KPIs do you use when the statistics are being used to measure whether the KPIs are the right ones?

The take away is that governments and large institutions which impact large numbers of people should never attempt to set quantifiable targets and should never attempt to meet quantifiable targets.

Jtsummers 5y ago

That we can't stop thinking. It becomes too easy to stop using our own judgement when we have numbers to back up our decisions. We can point to he analysis and say, "Look, we've improved <metric>!". That itself becomes the justification for the actions taken, regardless of their actual sensibility. This is an area where discipline remains essential, and maintaining discipline is a constant battle.

castlecrasher2 5y ago

That statistics should inform action or measure of efficacy, but not drive it. A dev manager running on only numbers will inevitably get empty, meaningless values such as tickets resolved or lines of code written, while the polar opposite manager will run on intuition alone. I imagine most would agree neither type is generally effective and a balance should be struck, and Goodhart's Law means you should be aware of what's important, pay attention to it, but do not make it your sole focus. And for God's sake, don't make a public dashboard for it.

ItsMonkk 5y ago

You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible. Not for everyone, anyway. Using hiring as an example, but this should work for nearly all metrics driven workloads. Hire a small subset of your staff using a real judgment system, hire most of your staff using metrics, then take a sample of those hired using metrics and take a real look at them compared against those hired using real judgments. If they are reasonably close, the KPIs are working. If they are alien to each-other, you need to either stop using KPIs or alter them significantly to fit with what the more effective people are doing.

rhizome 5y ago
>You need to measure the KPIs. What KPIs do you use when the statistics are being used to measure whether the KPIs are the right ones?