6 ms·

How a Machine Learns Prejudice

37 points by matthberg 10y ago

andrewclunn 10y ago
There seems to be a bit of explaining away all bias as the result of human creators here. If an algorithm trained on real world data displayed some traditional bias, would we assume such a cause and dismiss it? One could just as easily make this argument to dismiss all correlation based evidence. Yes the code / methodology matters, but we'd best get ready for deep learning algorithms to confront us with some uncomfortable truths.
- empath75 10y ago
  Let's assume you have red and green people who are otherwise identical, and that there was systemic discrimination against green people in the past that led them to have lower incomes and lower education levels and live in areas with lower property values and higher crime rates. A naive computer model trained on objective data might find correlations between green people and high default rates on loans, for example, which might be accurate, but which aren't inherent to green people but were forced on them by external circumstances. A reasonable bank might then conclude that they should discriminate against green people without any personal bias against green people at all. The question you have to ask yourself is: is that fair? And what should the law say about it?
  - iopq 10y ago
    They aren't inherent in green people, but you shouldn't give them loans anyway. Loans should be based on reality, not an ideal world that we don't live in. And certainly fairness has no basis in reality. It's a human concept.
  - steveeq1 10y ago
    (quote by some guy named "oehead2000" on some other hacker news posting. It fit the context perfectly, so I am copying and pasting from his reply) "Unfortunately whatever algorithm computers come up with to assess creditworthiness is going to be 'racist' by the broad disparate impact definition of racism that the government uses. Ironically, the algorithms will have to be made actually racist to correct for this."
  - andrewclunn 10y ago
    Most significant variable. If green is a proxy for poor, which is in turn a proxy for risky mortgage loan, then the poor variable is the better point of discrimination.
  - h4nkoslo 10y ago
    Well, the proper response probably isn't to shoot yourself in the head and prevent yourself from knowing about the difference in default rates. It is entirely possible to figure out what is going on, to what extent, and then make an informed decision as to whether to continue. Avoiding discovering things, which is what all these asinine articles end up suggesting, is almost never a good idea for competent adults, and legally preventing people from discovering things is antithetical to a free society.
  - atrendyguy 10y ago
    Forcing non-optimal decisions based on an arbitrary notion of "fairness" will only weaken our institutions and breed resentment: "group A gets loan approval with a lower credit score than members of my group, what gives!?" It also increases the fragility of the group you're trying to help by acclimating them to lower standards. The success of Asian and Jewish people suggests that things can correct themselves over time. The law should stay out of it. http://www.pewresearch.org/daily-number/asian-americans-lead-all-others-in-household-income/ http://www.pewresearch.org/daily-number/asian-americans-lead... http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chinese-persecution-in-the-USA-history-timeline.htm http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chine...
h4nkoslo 10y ago
There is literally no support in the article for the contention that "artificial intelligence picks up bias from human creators" as opposed to making correct inferences from reality. All of the examples they provide, modulo blurry tank pix, are of the latter.
deleted 10y ago
[deleted]
randyrand 10y ago
Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!
- ceejayoz 10y ago
  "The computer can't be wrong" is an equally dangerous attitude.
- jjoonathan 10y ago
  What makes me sad is that there are good reasons to throw out information which don't involve denying mathematical truth, yet people, for whatever reason, seem to prefer denial. What should happen: "I understand that this person has black skin and therefore according to elementary statistics is more likely to be a criminal, but acting on the information would create an unfair double standard that punishes people for crimes they didn't commit, it would help perpetuate a cycle, and it would violate individual human dignity, so I won't." What actually happens: exactly what parent said. "Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!" facepalm.
  - deleted 10y ago
    [deleted]
- Tharre 10y ago
  It's not quite like that. Though blaming algorithms for being "biased" is rather strange as well. It seems to me the algorithms do exactly what you'd expect them to: have predictive capabilities about reality. Of course, there is a problem here. You don't train neural networks with data of people who've committed crimes. You don't have such data. So what data has been used by the engineers to feed to this machine? Data of people who have been convicted of committing a crime, mostly likely. And now you've constructed a feedback loop. Those pesky blacks are more likely to commit crimes? Always knew it. A man has been murdered and there are 2 suspects, one white and one black? Why even bother investigating the white guy, we already knew it was the black one. Holy cow, 1 year with the new system and black crime has increased by 50%! Always knew it. Better keep the system in place to save everyone from those pesky blacks. Oh, the irony.
h4nkoslo 10y ago
The interesting thing is that literally all of the inferences we are "worried" about computers making were social consensuses not too long ago, with ample first-order evidence to support them even in the absence of fancy machine learning algorithms.
KODeKarnage 10y ago
> In June 2015, for example, Google’s photo categorization system identified two African Americans as “gorillas.” > Law enforcement officials have already been criticized, for example, for using computer algorithms that allegedly tag black defendants as more likely to commit a future crime, even though the program was not designed to explicitly consider race. Consider these two examples from the article. One is when the machine failed in a particular undesirable fashion. The other is when the machine "worked^" but in an undesirable fashion. (^That is, overall it might have worked well, but the bias in errors wasn't sufficiently explored.) Modelling "undesirable fashion", which is a dynamic, subjective social construct, is far more difficult than either of the tasks originally set in the examples. In the first example, how many images of people were confused with non-humans? I can't believe this was the only failed result. The only reason this particular example was problematic was that these specific people were specifically identified as gorillas. No problem if they were identified as a car, or a chess piece, or a satellite. Gorillas though, that's a special failure. We all know why. And we know why the error occurred. If the machine sees predominantly white people and gorillas, then "people" are white and "almost people" but with darker pixels is "gorilla". It is human prejudice in interpreting the results that made the error an issue.
kahrkunne 10y ago
"racist computers" is the funniest trend since "sexist babies" It should be obvious that if an ML algorithm is trained on sufficient real-world data, it won't be racist - it'll just not be politically correct
- arjie 10y ago
  Why is that obvious? Take the case of criminal predictions. If you're using convictions as ground truth, then you're just mirroring existing societal biases. For instance, in a society with pervasive racism (like, say 1960s America), the more real-world data you give it, the more your model is likely to converge on racism. Computers aren't racist but to characterise the article as referring to something like that is to fail to understand it.
  - KODeKarnage 10y ago
    > If you're using convictions as ground truth, then you're just mirroring existing societal biases. And if you don't use real life data...? What do you get?
  - randyrand 10y ago
    Not as bleak as you imply. Just don't give the model race or strong indicators of race as an input.
    - KODeKarnage 10y ago
      That's the problem. If there is ANY bias shown in your results, your algo (actually you) will be accused of prejudice. That is what happened with the Re-Offending Risk Assessment in the linked article. It didn't have race or anything like it. It had suburb.
      - jonathankoren 10y ago
        Please. Zip code is a well known proxy for race given the history of racist housing practices. Don't believe it? Ask yourself: What does a person from Harlem look like?
dominotw 10y ago
whats with shitty scientificamerican stories on HN today.
- grzm 10y ago
  If you don't think the submission is appropriate for HN, flag it if appropriate and move on.

37 points by matthberg 10y ago

There seems to be a bit of explaining away all bias as the result of human creators here. If an algorithm trained on real world data displayed some traditional bias, would we assume such a cause and dismiss it? One could just as easily make this argument to dismiss all correlation based evidence. Yes the code / methodology matters, but we'd best get ready for deep learning algorithms to confront us with some uncomfortable truths.

empath75 10y ago
Let's assume you have red and green people who are otherwise identical, and that there was systemic discrimination against green people in the past that led them to have lower incomes and lower education levels and live in areas with lower property values and higher crime rates. A naive computer model trained on objective data might find correlations between green people and high default rates on loans, for example, which might be accurate, but which aren't inherent to green people but were forced on them by external circumstances. A reasonable bank might then conclude that they should discriminate against green people without any personal bias against green people at all. The question you have to ask yourself is: is that fair? And what should the law say about it?
- iopq 10y ago
  They aren't inherent in green people, but you shouldn't give them loans anyway. Loans should be based on reality, not an ideal world that we don't live in. And certainly fairness has no basis in reality. It's a human concept.
- steveeq1 10y ago
  (quote by some guy named "oehead2000" on some other hacker news posting. It fit the context perfectly, so I am copying and pasting from his reply) "Unfortunately whatever algorithm computers come up with to assess creditworthiness is going to be 'racist' by the broad disparate impact definition of racism that the government uses. Ironically, the algorithms will have to be made actually racist to correct for this."
- andrewclunn 10y ago
  Most significant variable. If green is a proxy for poor, which is in turn a proxy for risky mortgage loan, then the poor variable is the better point of discrimination.
- h4nkoslo 10y ago
  Well, the proper response probably isn't to shoot yourself in the head and prevent yourself from knowing about the difference in default rates. It is entirely possible to figure out what is going on, to what extent, and then make an informed decision as to whether to continue. Avoiding discovering things, which is what all these asinine articles end up suggesting, is almost never a good idea for competent adults, and legally preventing people from discovering things is antithetical to a free society.
- atrendyguy 10y ago
  Forcing non-optimal decisions based on an arbitrary notion of "fairness" will only weaken our institutions and breed resentment: "group A gets loan approval with a lower credit score than members of my group, what gives!?" It also increases the fragility of the group you're trying to help by acclimating them to lower standards. The success of Asian and Jewish people suggests that things can correct themselves over time. The law should stay out of it. http://www.pewresearch.org/daily-number/asian-americans-lead-all-others-in-household-income/ http://www.pewresearch.org/daily-number/asian-americans-lead... http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chinese-persecution-in-the-USA-history-timeline.htm http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chine...

h4nkoslo 10y ago

There is literally no support in the article for the contention that "artificial intelligence picks up bias from human creators" as opposed to making correct inferences from reality. All of the examples they provide, modulo blurry tank pix, are of the latter.

deleted 10y ago

[deleted]

randyrand 10y ago

Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!

ceejayoz 10y ago
"The computer can't be wrong" is an equally dangerous attitude.
jjoonathan 10y ago
What makes me sad is that there are good reasons to throw out information which don't involve denying mathematical truth, yet people, for whatever reason, seem to prefer denial. What should happen: "I understand that this person has black skin and therefore according to elementary statistics is more likely to be a criminal, but acting on the information would create an unfair double standard that punishes people for crimes they didn't commit, it would help perpetuate a cycle, and it would violate individual human dignity, so I won't." What actually happens: exactly what parent said. "Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!" facepalm.
- deleted 10y ago
  [deleted]
Tharre 10y ago
It's not quite like that. Though blaming algorithms for being "biased" is rather strange as well. It seems to me the algorithms do exactly what you'd expect them to: have predictive capabilities about reality. Of course, there is a problem here. You don't train neural networks with data of people who've committed crimes. You don't have such data. So what data has been used by the engineers to feed to this machine? Data of people who have been convicted of committing a crime, mostly likely. And now you've constructed a feedback loop. Those pesky blacks are more likely to commit crimes? Always knew it. A man has been murdered and there are 2 suspects, one white and one black? Why even bother investigating the white guy, we already knew it was the black one. Holy cow, 1 year with the new system and black crime has increased by 50%! Always knew it. Better keep the system in place to save everyone from those pesky blacks. Oh, the irony.

h4nkoslo 10y ago

The interesting thing is that literally all of the inferences we are "worried" about computers making were social consensuses not too long ago, with ample first-order evidence to support them even in the absence of fancy machine learning algorithms.

KODeKarnage 10y ago

> In June 2015, for example, Google’s photo categorization system identified two African Americans as “gorillas.” > Law enforcement officials have already been criticized, for example, for using computer algorithms that allegedly tag black defendants as more likely to commit a future crime, even though the program was not designed to explicitly consider race. Consider these two examples from the article. One is when the machine failed in a particular undesirable fashion. The other is when the machine "worked^" but in an undesirable fashion. (^That is, overall it might have worked well, but the bias in errors wasn't sufficiently explored.) Modelling "undesirable fashion", which is a dynamic, subjective social construct, is far more difficult than either of the tasks originally set in the examples. In the first example, how many images of people were confused with non-humans? I can't believe this was the only failed result. The only reason this particular example was problematic was that these specific people were specifically identified as gorillas. No problem if they were identified as a car, or a chess piece, or a satellite. Gorillas though, that's a special failure. We all know why. And we know why the error occurred. If the machine sees predominantly white people and gorillas, then "people" are white and "almost people" but with darker pixels is "gorilla". It is human prejudice in interpreting the results that made the error an issue.

kahrkunne 10y ago

"racist computers" is the funniest trend since "sexist babies" It should be obvious that if an ML algorithm is trained on sufficient real-world data, it won't be racist - it'll just not be politically correct

arjie 10y ago
Why is that obvious? Take the case of criminal predictions. If you're using convictions as ground truth, then you're just mirroring existing societal biases. For instance, in a society with pervasive racism (like, say 1960s America), the more real-world data you give it, the more your model is likely to converge on racism. Computers aren't racist but to characterise the article as referring to something like that is to fail to understand it.
- KODeKarnage 10y ago
  > If you're using convictions as ground truth, then you're just mirroring existing societal biases. And if you don't use real life data...? What do you get?
- randyrand 10y ago
  Not as bleak as you imply. Just don't give the model race or strong indicators of race as an input.
  - KODeKarnage 10y ago
    That's the problem. If there is ANY bias shown in your results, your algo (actually you) will be accused of prejudice. That is what happened with the Re-Offending Risk Assessment in the linked article. It didn't have race or anything like it. It had suburb.
    - jonathankoren 10y ago
      Please. Zip code is a well known proxy for race given the history of racist housing practices. Don't believe it? Ask yourself: What does a person from Harlem look like?

dominotw 10y ago

whats with shitty scientificamerican stories on HN today.

grzm 10y ago
If you don't think the submission is appropriate for HN, flag it if appropriate and move on.