3 ms·

Every time I see an article like this about how the UK government (all large orginzations of people in general though) is obssessed with measuring meaningless t

Every time I see an article like this about how the UK government (all large orginzations of people in general though) is obssessed with measuring meaningless things, I am reminded that it is the birthplace of Goodhart's Law [0]. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

IIAOPSW 6y ago
But the alternative to decision by metrics is decision by committee. The former may be gamed, but the latter may be corrupt. The inherent human nature that leads to corruption cannot be fixed, but in theory there could be a metric that isn't flawed. Therefore the solution is not to abandon evidence based policy, but to have more and better evidence based policy.
- caddemon 6y ago
  I've always wondered about using multiple metrics, but instead of combining them prior to selection, bin the spots and for each bin consider a subset of metrics weighted in different ways. For something like a school ranking, instead of giving a number (which is way more precision than I think these systems can actually achieve), the schools could be tiered - with each bin contributing some spots to the first tier, then repeat. At the very least it should diversify how people try to game the system, and then the "winners" of various bins could be studied to compare usage of different metrics in selection. If the specific formula could be kept mostly secret, perhaps switched up every so often, it would hopefully disincentivize gaming at all. Along those lines, committees and metrics could both be used. Give committees control over 10% of the spots or something. Corruption is obviously a downside, but I do think committees also have upsides that are difficult if not impossible to replicate with pure metrics. In doing this committee selections could also be tracked against the various metric ones for longer term outcomes.
iheartblocks 6y ago
I'm currently watching The Wire, and this aspect of people juking the stats to move up the ranks is a deep underlying theme. It's gotten me to thinking: what is the best alternative to focusing on metrics? Surely, relying on something other than data would be no improvement over the status quo. The only thought I've had is to: 1) Rely on a spectrum of metrics, to get a more holistic perspective. Instead of just seeing "crime is down x%" administrators would see how many reports end up classified as a lower-level infractions, how civilians rate the safety of their area, etc. 2) Add common-sense restrictions to make stat-pumping more difficult. For example, bringing in classroom inspections on random days 3) Penalizing institutions and individuals who are clearly attempting to juke the stats. Enforcement would certainly be difficult, but simply having something to this effect on the books would, imho, reduce schools dedicating time to studying for a standardized test

IIAOPSW 6y ago

But the alternative to decision by metrics is decision by committee. The former may be gamed, but the latter may be corrupt. The inherent human nature that leads to corruption cannot be fixed, but in theory there could be a metric that isn't flawed. Therefore the solution is not to abandon evidence based policy, but to have more and better evidence based policy.

caddemon 6y ago
I've always wondered about using multiple metrics, but instead of combining them prior to selection, bin the spots and for each bin consider a subset of metrics weighted in different ways. For something like a school ranking, instead of giving a number (which is way more precision than I think these systems can actually achieve), the schools could be tiered - with each bin contributing some spots to the first tier, then repeat. At the very least it should diversify how people try to game the system, and then the "winners" of various bins could be studied to compare usage of different metrics in selection. If the specific formula could be kept mostly secret, perhaps switched up every so often, it would hopefully disincentivize gaming at all. Along those lines, committees and metrics could both be used. Give committees control over 10% of the spots or something. Corruption is obviously a downside, but I do think committees also have upsides that are difficult if not impossible to replicate with pure metrics. In doing this committee selections could also be tracked against the various metric ones for longer term outcomes.

iheartblocks 6y ago

I'm currently watching The Wire, and this aspect of people juking the stats to move up the ranks is a deep underlying theme. It's gotten me to thinking: what is the best alternative to focusing on metrics? Surely, relying on something other than data would be no improvement over the status quo. The only thought I've had is to: 1) Rely on a spectrum of metrics, to get a more holistic perspective. Instead of just seeing "crime is down x%" administrators would see how many reports end up classified as a lower-level infractions, how civilians rate the safety of their area, etc. 2) Add common-sense restrictions to make stat-pumping more difficult. For example, bringing in classroom inspections on random days 3) Penalizing institutions and individuals who are clearly attempting to juke the stats. Enforcement would certainly be difficult, but simply having something to this effect on the books would, imho, reduce schools dedicating time to studying for a standardized test