3 ms·

Consider your drug conviction example, and lets say the algorithm is linear regression so I can give a very simple explanation. The model will then be FICO = s

Consider your drug conviction example, and lets say the algorithm is linear regression so I can give a very simple explanation. The model will then be FICO = stuff + (-1) x (drug conviction). I.e., your FICO score is lowered by 1 point if you've been convicted of drug use. If you are correct, some enterprising quant can take your hypothesis. He can then rerun his regression with (drug conviction, black) and (drug conviction, white) as variables. If your hypothesis is correct the result will be: FICO = stuff + (-0.5) x (drug conviction, black) + (-1.5) x (drug conviction, white). This is because the bias in drug convictions makes it less predictive for black people. If you are wrong the coefficients will be equal [1]. If you are right he just made millions of dollars for the bank he works for and probably added $100k to his bonus. What you are discussing is strictly a statistics problem. [1] For examples of this type of analysis on social indicators (I don't know of any on credit allocation), see these papers: https://www.rand.org/content/dam/rand/www/external/labor/seminars/adp/pdfs/adp_ajph.pdf https://www.rand.org/content/dam/rand/www/external/labor/sem... http://egov.ufsc.br/portal/sites/default/files/anexos/33027-41458-1-PB.pdf http://egov.ufsc.br/portal/sites/default/files/anexos/33027-... http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf

pron 11y ago
The point is that assigning a FICO score -- any score -- is a strong predictor of future scores, to the person and those dependent on them. If the current score is unfairly distributed in society due to other historical effects, the computation will actively work to maintain the distribution. A quant doesn't care how they make their money, and they certainly have no desire in leaving their Nash equilibrium. Again, if you say you want to enact a conservative policy, say so. It's perfectly OK. But don't program conservatism into an algorithm and then claim "it's only math". It isn't. Of course, being in a Nash equilibrium makes conservative algorithms much easier, as they don't require bargaining with other players, but that doesn't make them any less conservative.
- AnthonyMouse 11y ago
  Suppose we have some algorithms which are trying to predict which team will win a game. People are going to bet money based on what the algorithms say, and teams benefit when more money is bet on them. The red team plays dirty. First they spread false rumours about the blue team. Most of the algorithms respond by reducing the blue team's chances of winning against the red team, but algorithm #5 successfully predicts that the rumours were false and people who bet using algorithm #5 made a lot of money. Now lots of people use variants of algorithm #5 and thereafter the red team has no luck getting anyone to believe their false rumours. In other words, if the algorithms are incorrect then we're all better off to fix them. Next the red team successfully lobbies the government to raise the blue team's taxes (even though the red team makes more money). Now the blue team has less money for training and equipment, which has actually reduced their chances of winning the game. Most of the algorithms successfully predict this, and then the blue team loses the next game. Your argument seems to be that the red team should be punished for this by having your algorithm stop preferring them. But now you're screwing over everyone else relying on your algorithm, who weren't responsible for the behaviour you're trying to punish. And thereafter no one trusts you to provide true information. If the red team is playing dirty and the math accurately predicts the result, the problem is not the math.
  - drdeca 11y ago
    do you mean red team / blue team as in attack / defense or as in red tribe / blue tribe? or just names for teams?
  - pron 11y ago
    Your analogy is incorrect because when it comes to social systems, how people bet actually changes the outcome, at least in the long run. So the dynamics are more like that the team that wins is the one that has the most people bet it would win. In general, the problem is not the math but our interpretation of the result. All the result says is that if we change nothing, this is the likely outcome. By interpreting the results to mean merely "this is the likely outcome", we are depriving people of the opportunity of getting out of local optima. Deciding to coordinate or deciding to have society maintain its Nash equilibria are both political choices -- not mathematical outcomes.
AnthonyMouse 11y ago
You're essentially making an argument from the efficient market hypothesis: https://en.wikipedia.org/wiki/Efficient-market_hypothesis https://en.wikipedia.org/wiki/Efficient-market_hypothesis In other words, the algorithm won't be biased because there is a buck to be made by removing the bias. But the efficient market hypothesis is weird. It's kind of like Schrodinger's Cat. If you can find some specific instance where it's wrong and publish it then the market corrects itself, so if you observe a bias then it ceases to exist. So now on one hand, we know that biases can exist because people have discovered them in the past; it's just that those biases have already been published and are being taken into account. We can speculate that more exist but we don't know what they are. On the other hand, maybe now we've finally found them all (or the remaining ones are negligible) and the efficient market hypothesis becomes true. But notice that the thing to do if you believe the efficient market hypothesis is wrong in a particular case is clear. Prove it, because you'll be able to make a lot of money by making the world more just. Which is why we need transparency. Because the market participants can't find the bias if they don't have the information, and then the efficient market hypothesis will certainly be false.
- yummyfajitas 11y ago
  I made no claims about the EMH. My only claim is that if bias like what you describe exists, it's something that statisticians rather than politicians need to identify and fix, and that the statisticians in the position to fix it have perfect incentives to do so. Politicians have neither the ability nor incentive. Similarly, if someone were claiming that we need political oversight of UI/UX choices, and that American web pages need more red/white/blue to improve conversion rates, I'd suggest that this is a job for web designers rather than politicians. I'd also say that if you just want red/white/blue designs for patriotic purposes, you should openly advocate for that and not talk about conversion rate.

pron 11y ago

The point is that assigning a FICO score -- any score -- is a strong predictor of future scores, to the person and those dependent on them. If the current score is unfairly distributed in society due to other historical effects, the computation will actively work to maintain the distribution. A quant doesn't care how they make their money, and they certainly have no desire in leaving their Nash equilibrium. Again, if you say you want to enact a conservative policy, say so. It's perfectly OK. But don't program conservatism into an algorithm and then claim "it's only math". It isn't. Of course, being in a Nash equilibrium makes conservative algorithms much easier, as they don't require bargaining with other players, but that doesn't make them any less conservative.

AnthonyMouse 11y ago
Suppose we have some algorithms which are trying to predict which team will win a game. People are going to bet money based on what the algorithms say, and teams benefit when more money is bet on them. The red team plays dirty. First they spread false rumours about the blue team. Most of the algorithms respond by reducing the blue team's chances of winning against the red team, but algorithm #5 successfully predicts that the rumours were false and people who bet using algorithm #5 made a lot of money. Now lots of people use variants of algorithm #5 and thereafter the red team has no luck getting anyone to believe their false rumours. In other words, if the algorithms are incorrect then we're all better off to fix them. Next the red team successfully lobbies the government to raise the blue team's taxes (even though the red team makes more money). Now the blue team has less money for training and equipment, which has actually reduced their chances of winning the game. Most of the algorithms successfully predict this, and then the blue team loses the next game. Your argument seems to be that the red team should be punished for this by having your algorithm stop preferring them. But now you're screwing over everyone else relying on your algorithm, who weren't responsible for the behaviour you're trying to punish. And thereafter no one trusts you to provide true information. If the red team is playing dirty and the math accurately predicts the result, the problem is not the math.
- drdeca 11y ago
  do you mean red team / blue team as in attack / defense or as in red tribe / blue tribe? or just names for teams?
- pron 11y ago
  Your analogy is incorrect because when it comes to social systems, how people bet actually changes the outcome, at least in the long run. So the dynamics are more like that the team that wins is the one that has the most people bet it would win. In general, the problem is not the math but our interpretation of the result. All the result says is that if we change nothing, this is the likely outcome. By interpreting the results to mean merely "this is the likely outcome", we are depriving people of the opportunity of getting out of local optima. Deciding to coordinate or deciding to have society maintain its Nash equilibria are both political choices -- not mathematical outcomes.

AnthonyMouse 11y ago

You're essentially making an argument from the efficient market hypothesis: https://en.wikipedia.org/wiki/Efficient-market_hypothesis https://en.wikipedia.org/wiki/Efficient-market_hypothesis In other words, the algorithm won't be biased because there is a buck to be made by removing the bias. But the efficient market hypothesis is weird. It's kind of like Schrodinger's Cat. If you can find some specific instance where it's wrong and publish it then the market corrects itself, so if you observe a bias then it ceases to exist. So now on one hand, we know that biases can exist because people have discovered them in the past; it's just that those biases have already been published and are being taken into account. We can speculate that more exist but we don't know what they are. On the other hand, maybe now we've finally found them all (or the remaining ones are negligible) and the efficient market hypothesis becomes true. But notice that the thing to do if you believe the efficient market hypothesis is wrong in a particular case is clear. Prove it, because you'll be able to make a lot of money by making the world more just. Which is why we need transparency. Because the market participants can't find the bias if they don't have the information, and then the efficient market hypothesis will certainly be false.

yummyfajitas 11y ago
I made no claims about the EMH. My only claim is that if bias like what you describe exists, it's something that statisticians rather than politicians need to identify and fix, and that the statisticians in the position to fix it have perfect incentives to do so. Politicians have neither the ability nor incentive. Similarly, if someone were claiming that we need political oversight of UI/UX choices, and that American web pages need more red/white/blue to improve conversion rates, I'd suggest that this is a job for web designers rather than politicians. I'd also say that if you just want red/white/blue designs for patriotic purposes, you should openly advocate for that and not talk about conversion rate.