4 ms·
> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors.
by AnthonyMouse 11y ago
> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above.
There is a legitimate question as to whether an algorithm is actually unbiased. It could be quite accurate or there could be a bias, e.g. because the algorithm penalizes drug convictions and drug convictions have an unjustified bias due to racist or sexist enforcement.
The problem is it isn't a simple question to answer. And people are going to want to choose their answer based on their politics.
But that can't work. We can't let a black box take discriminatory inputs and expect it to produce a just result. We also can't just assume that every unequal outcome had an unjust cause, or we'll end up e.g. causing loans to be given to uncreditworthy people and wrecking the economy (see also housing crash).
Which means that Schneier is wrong about quotas but exactly right that what we need is transparency. Because if there is unjustified bias then we need to understand why so we can prevent it, and if there isn't then we need to understand why so we can accept it.
- yummyfajitas 11y agoConsider your drug conviction example, and lets say the algorithm is linear regression so I can give a very simple explanation. The model will then be FICO = stuff + (-1) x (drug conviction). I.e., your FICO score is lowered by 1 point if you've been convicted of drug use. If you are correct, some enterprising quant can take your hypothesis. He can then rerun his regression with (drug conviction, black) and (drug conviction, white) as variables. If your hypothesis is correct the result will be: FICO = stuff + (-0.5) x (drug conviction, black) + (-1.5) x (drug conviction, white). This is because the bias in drug convictions makes it less predictive for black people. If you are wrong the coefficients will be equal [1]. If you are right he just made millions of dollars for the bank he works for and probably added $100k to his bonus. What you are discussing is strictly a statistics problem. [1] For examples of this type of analysis on social indicators (I don't know of any on credit allocation), see these papers: https://www.rand.org/content/dam/rand/www/external/labor/seminars/adp/pdfs/adp_ajph.pdf https://www.rand.org/content/dam/rand/www/external/labor/sem... http://egov.ufsc.br/portal/sites/default/files/anexos/33027-41458-1-PB.pdf http://egov.ufsc.br/portal/sites/default/files/anexos/33027-... http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf
- pron 11y agoThe point is that assigning a FICO score -- any score -- is a strong predictor of future scores, to the person and those dependent on them. If the current score is unfairly distributed in society due to other historical effects, the computation will actively work to maintain the distribution. A quant doesn't care how they make their money, and they certainly have no desire in leaving their Nash equilibrium. Again, if you say you want to enact a conservative policy, say so. It's perfectly OK. But don't program conservatism into an algorithm and then claim "it's only math". It isn't. Of course, being in a Nash equilibrium makes conservative algorithms much easier, as they don't require bargaining with other players, but that doesn't make them any less conservative.
- AnthonyMouse 11y agoSuppose we have some algorithms which are trying to predict which team will win a game. People are going to bet money based on what the algorithms say, and teams benefit when more money is bet on them. The red team plays dirty. First they spread false rumours about the blue team. Most of the algorithms respond by reducing the blue team's chances of winning against the red team, but algorithm #5 successfully predicts that the rumours were false and people who bet using algorithm #5 made a lot of money. Now lots of people use variants of algorithm #5 and thereafter the red team has no luck getting anyone to believe their false rumours. In other words, if the algorithms are incorrect then we're all better off to fix them. Next the red team successfully lobbies the government to raise the blue team's taxes (even though the red team makes more money). Now the blue team has less money for training and equipment, which has actually reduced their chances of winning the game. Most of the algorithms successfully predict this, and then the blue team loses the next game. Your argument seems to be that the red team should be punished for this by having your algorithm stop preferring them. But now you're screwing over everyone else relying on your algorithm, who weren't responsible for the behaviour you're trying to punish. And thereafter no one trusts you to provide true information. If the red team is playing dirty and the math accurately predicts the result, the problem is not the math.
- drdeca 11y ago
- danmaz74 11y ago> Which means that Schneier is wrong about quotas I'm not a big fan of quotas in general, don't think they are the end-all solution to discrimination problems, and know they can look like discrimination themselves. But there is an interesting point of view to consider. When we use these kinds of algorithms to optimize some wanted result, we are always applying some form of reinforcement learning [1]. Reinforcement learning always needs to balance exploration and exploitation of our current knowledge, and quotas are one socially justifiable way to increase exploration - and not get stuck in self-reinforcing local maximum with a positive feedback cycle. So, even from a purely utilitarian point of view, a certain amount of reshuffling of the cards (eg with quotas, but not limited to those) can be useful. [1] https://en.wikipedia.org/wiki/Reinforcement_learning https://en.wikipedia.org/wiki/Reinforcement_learning