4 ms·
I've always wondered about using multiple metrics, but instead of combining them prior to selection, bin the spots and for each bin consider a subset of metrics
by caddemon 6y ago
I've always wondered about using multiple metrics, but instead of combining them prior to selection, bin the spots and for each bin consider a subset of metrics weighted in different ways. For something like a school ranking, instead of giving a number (which is way more precision than I think these systems can actually achieve), the schools could be tiered - with each bin contributing some spots to the first tier, then repeat.
At the very least it should diversify how people try to game the system, and then the "winners" of various bins could be studied to compare usage of different metrics in selection. If the specific formula could be kept mostly secret, perhaps switched up every so often, it would hopefully disincentivize gaming at all.
Along those lines, committees and metrics could both be used. Give committees control over 10% of the spots or something. Corruption is obviously a downside, but I do think committees also have upsides that are difficult if not impossible to replicate with pure metrics. In doing this committee selections could also be tracked against the various metric ones for longer term outcomes.