6 ms·
And the number will get gamed.
by kiba 11y ago
And the number will get gamed.
- astazangasta 11y agoThis comment could be attached to any proposed system of accountability based on a simple metric.
- knowaveragejoe 11y agoSo what do we do about it?
- TeMPOraL 11y agoShoot people who game the system. Seriously, there is no solution to the problem of humans abusing systems that is not GAI-complete. We can't design solutions that strong, but we ourselves are that strong. We have a pretty good solution available, that we don't use as often as we should: recognize and punish assholes (or, more formally, people who defect in coordination problems).
- ericflo 11y agoWhat's GAI-complete? I'm assuming you don't mean getaddrinfo and a Google search doesn't reveal much.
- dllthomas 11y agoI assume GAI is "General Artificial Intelligence" - so that "GAI-complete" is "at least as hard as producing a general artificial intelligence."
- TeMPOraL 11y agoYou assume correctly.
- TeMPOraL 11y agoWhat 'dllthomas said. I'm sorry, I should have explained in the comment, it's an obscure term to use.
- AndrewKemendo 11y agoGAI complete is a term used by people who don't actually work in AI and assume that their use of GAI is somehow equivalent to AGI, which is the term of art for Strong AI - codified with the AGI Journal and AGI conference.
- bitJericho 11y agoIn the real work we have systems of checks and balances. Simply hire a team of compliance checkers. Those checkers are kept in line by their management. If management is not up to task it'll come out eventually in a fabulous scandal, but in the mean time it'd be better than nothing.
- TeMPOraL 11y agoCompliance checking is a formal way of identifying and firing assholes. As for "checks and balances", yes, separating powers between several groups of people to enforce coordination between them is a good thing, but it also gets gamed at a meta-level, which again means you need to find and punish meta-level assholes.
- eru 11y agoActually, simpler metrics are better. It's more obvious, when you game them.
- astazangasta 11y agoDesign systems that rely on human conversation instead.
- detaro 11y agoSo no difference to the status quo in that regard.
- hyperion2010 11y agoExcept that one way to game recidivism rates is to kill them.
- jessaustin 11y agoFatality rates seem more difficult to game than recidivism rates.
- AnkhMorporkian 11y agoAnd one way to improve standardized test scores would be killing low-achieving students, but you don't see schools doing that despite the funding incentive. Even ignoring the moral issue, after all most prison administrators are indeed human, there's just no way to game it without getting caught. You might take out a few, but that would barely impact recidivism rates at any prison. We're talking about thousands of prisoners here, and a recidivism rate that hovers around 75%. That'd be hundreds of deaths to meaningfully impact a prison's recidivism rate. I don't care how powerful you are, you aren't going to kill hundreds of people all linked by a very identifiable thread and get away with it.
- function_seven 11y ago> And one way to improve standardized test scores would be killing low-achieving students, but you don't see schools doing that despite the funding incentive. No, but they may be doing something similar[1]: Schools at times suspend, expel, "counsel out" or otherwise remove students with low scores in order to boost school results and escape test-based sanctions mandated by the federal government's "No Child Left Behind" law, at great cost to the youth and ultimately society. I don't know the veracity of the linked paper, but it's certainly plausible and I'm sure it has occurred somewhere. If you're under enormous pressure to raise test scores, and your district has a continuation school, then it seems like the lowest-scoring students could be shuffled off to that school and the average increased. Not that I'm making a value judgement here. It's possible that doing this is actually the best course of action in many instances, but it's also possible the the transfer technique could be abused to the detriment of some students. All in all, it's extremely hard to develop a metric that's resistant to gaming and manipulation. EDIT: Here's another good link. More of a primary source[2] [1](PDF) http://www.fairtest.org/sites/default/files/racial_justice_and_testing_12-10.pdf http://www.fairtest.org/sites/default/files/racial_justice_a... [2] http://www.nber.org/papers/w11194 http://www.nber.org/papers/w11194
- SixSigma 11y agoWell, in business you set up two competing KPIs so that gaming is harder. Single metrics are rarely successful.
- SixSigma 11y agoReplying to myself: Setting drivers MPG as a kpi - best way to do that is not go anywhere. One of the tragedies of the 1980s was the corporate raider. The quickest way to look profitable is to slash costs. The raiders did this buy shutting down plants. Balance sheets looked good, stock price goes up, sell off the company shares, company dies because of no production. Profit of a company is really a poor metric for society. If 100,000 workers manage to build products but the company only breaks even is that a successful business?
- mabbo 11y agoAs someone else said, use multiple KPIs that describe an ideal individual who is reformed. Recidivism rate, average annual income over the next 10 years, health/estimated life-span of the reformed individual. Will prisons try to find ways to cheat? That's fine, we'll just reserve the right to redefine the KPIs every year for all new prisoners. But making such a change is difficult, because of "tough on crime" politicians who believe the only way to reduce crime is harsh justice- when in modern times that simply isn't true any more.
- bsder 11y agoIt would at least take them a couple of years to figure out how to game it optimally.
- Terr_ 11y agoUltimately everything's being gamed, the question is whether we'd collectively prefer a different equilibrium between the gaming parties.
- bshanks 11y agoPaying prisons based on gamed recidivism rates would probably still be better than not.