4 ms·
> The only solution I can see is forcing any company that imposes life-defining actions on people (credit bureaus, banks, parole boards, personnel offices, etc)
by AnthonyMouse 9y ago
> The only solution I can see is forcing any company that imposes life-defining actions on people (credit bureaus, banks, parole boards, personnel offices, etc) to use only rules based on objective criteria and to prohibit systems based on a "lasagna" of ad-hoc data like present day AI systems.
That is probably the exact opposite of what you really want. If the problem is that someone's name is Bob and the AI thinks Bobs are evil, what you want is for there to be 100,000 other factors for Bob to show the system that it isn't so. As many factors as possible, so that the one it gets wrong will have a very low weight.
Even the objective criteria will have biases. There is a significant racial disparity in prior criminal convictions, income, credit history and nearly every other "objective" factor. The more factors you bring in, the more opportunities someone in a given demographic has to prove they still deserve a chance.
- joe_the_user 9y agoThat is probably the exact opposite of what you really want. If the problem is that someone's name is Bob and the AI thinks Bobs are evil, what you want is for there to be 100,000 other factors for Bob to show the system that it isn't so, as many factors as possible, so that the one it gets wrong will have a very low weight. You don't understand. My point is that institutions making such decision should not be able to make decisions according to these 100,000 unexplained factors. If you're a lender, you can look at employment history, records of payment and other objective related criteria. You can't look at, say, eye color, however useful you might think it is. Institutions should be able make these decisions arbitrarily, at the level that they effect lives. There should legal provisions for auditing these things (as there are, on occasions, provisions of auditing affirmative action, environmental protection behaviors, insurance decisions, etc).
- AnthonyMouse 9y ago> My point is that institutions making such decision should not be able to make decisions according to these 100,000 unexplained factors. But how does that help anything? The objective factors have the same potential for bias as the seemingly irrelevant ones. All you get by excluding factors is to increase bias by not considering information that could mitigate the bias in the factors you are considering. Suppose that 80% of black men would be rejected for a loan based on your preferred set of objective factors. Of that 80%, many would actually repay the loan, but none of the objective factors can distinguish them from those who wouldn't, and when averaged together the outcome is to refuse the loan. If you used some seemingly arbitrary factors that happen to correlate for unknown reasons, you could profitably make loans to 60% of them instead of 20%. How is it helping anyone to not do that?
- AstralStorm 9y agoThe whole problem is that single cases are not statistics yet people would live to apply global generalized statistics to single cases. What you touched upon is the accuracy/ bias trade-off. To have evidence in particular case you need to attempt to debias the particular system and see how it affects accuracy. Sometimes, it may even vastly improve it. What is more important is that the systems are not benchmarked properly. As in compared against very simple metrics and systems. Such as: against random decision. Against simple recidivism prevention (grudge system). Against plain math metrics with constants. To add, they're opaque and it is impossible to easily extract the factors that went into any given single decision. This means they act fully irrationally. Intelligently but irrationally.
- im3w1l 9y agoLet's assume we push it from 20% to 23% (I don't think we can expect the huge gains you posted) by using various weird features such as whether you like to purchase odd or even number of bananas. People's live will depend on the decisions of these machines so people will start trying to game them. They will make sure to always purchase odd number of bananas, they will wear hats but only on Thursdays etc etc. Now two things happen. As more people game the system the rules need to be updated. Suddenly it's all about buying bananas divisible by three and wearing hats on weekends. The people who tried to follow the previous advice got screwed over, and what's more they have nothing to show for it. Instead of making people do useful things like paying bills on time and saving up some money, it made them follow some weird algorithmic fashion. Because of this expenditure of energy on meaningless things we may see that now only 18% of people would manage to pay back loans on time.
- teddyh 9y agoGoodhart’s law, as phrased by Marilyn Strathern: “When a measure becomes a target, it ceases to be a good measure.”
- AnthonyMouse 9y ago> People's live will depend on the decisions of these machines so people will start trying to game them. They will make sure to always purchase odd number of bananas, they will wear hats but only on Thursdays etc etc. But that's just another reason to use 100,000 factors instead of twelve. If someone's income is a huge factor in everything, people will spend more than the optimal amount of time working (instead of tending to their family or doing community service etc.), or choose to be high paid sleazy ambulance chasers instead of low paid teachers, because the algorithm makes the returns disproportionate to the effort. If buying an odd number of bananas is a factor but the effort of learning that it's a factor and then following it is larger than the expected benefit from changing one factor out of thousands, they won't do any of it.
- ori_b 9y ago> That is probably the exact opposite of what you really want. No, it really isn't. In an ideal world, the reasons behind a decision are transparent, auditable, understandable, and appealable. Machine learning is none of those.
- erik_seaberg 9y agoIn an ideal world, society would compensate you when they would like you to assume a risk below its market price, rather than forcing you to pretend not to notice the risk.
- joshuamorton 9y agoBut there isn't actually a risk associated with an Adolf. Its an inefficiency born out of an incorrect belief by the whole of (or at least most of) society. The correct solution is not to price in the incorrect assumption, but to not make the incorrect assumption. In other words, by offering Adolf's below-market rates, you're exploiting a market inefficiency at no additional risk. This is an ideal world as you describe it. It's capitalism at it's finest!
- dhbradshaw 9y agoThe interesting question is not whether those are good things -- they clearly are. The interesting question is their value relative to each other and, more specifically, relative to correctness. To what extent are we willing to tolerate less correct decisions if necessary in order to achieve those and other desirable properties? It seems like the answer to that question is situationally dependent.