4 ms·

Big data somehow being unbiased and impartial is a myth. The algorithms are still a product of humans. One notable example is a Google algorithm for labeling ob

by muraiki 10y ago

Big data somehow being unbiased and impartial is a myth. The algorithms are still a product of humans. One notable example is a Google algorithm for labeling objects in photos that tagged black people as gorillas [1]. Now I don't have particular insight into the development process behind this algorithm, but this result may have occurred because the training data used did not include enough photos of black people. Another example (from an old version of Picasa -- I'm not trying to pick on Google but I have firsthand experience of this) is when a friend of mine had photos of two different Asian people in his collection, but Picasa always recognized person B as being person A, presumably because it couldn't distinguish features unique to the faces of Asian people. [1] http://www.cnet.com/news/google-apologizes-for-algorithm-mistakenly-calling-black-people-gorillas/ http://www.cnet.com/news/google-apologizes-for-algorithm-mis...

yummyfajitas 10y ago
What evidence do you have that one specific classification error resulting in a tweetstorm was the result of bias at all? I've been mislabeled as a tank in at least one image processing demo (admittedly, this was well before the era of deep learning). Image classification errors happen. Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented among people building such algos, right? Consider the possibility that some classification problems are actually more difficult than others. As one example, darker objects are harder to distinguish than bright ones and there are very clear mathematical reasons for that. That's true even in fields where questions of "racism" don't make sense, e.g. in MRI (magnetic resonance imaging is non-optical, but contrast still matters): https://www.chrisstucchio.com/pubs/wf_segment.pdf https://www.chrisstucchio.com/pubs/wf_segment.pdf
- muraiki 10y ago
  I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning). As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be. Another example I heard recently from a conference was that of analyzing blighted homes in areas. The underlying data was generated by human inspectors who turned out to have given passes to violations in more affluent areas and fines in less affluent areas. The fines compounded the problem of blight as the residents could not afford to pay them and ended up causing a feedback loop of blight increasing in neighborhoods identified as blighted. I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. I apologize if I have only added confusion rather than a productive analysis.
  - yummyfajitas 10y ago
    I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning). But that's not true. Inadequate training data will cause variance, not bias. This variance can have any direction. In a linear prediction scenario (i.e. any go/no go decision process), which exactly what Cathy O'Neil is criticizing, the effect could just as easily be "more loans for black people" as "less loans for black people". In a multidimensional predictor, it does directly decrease accuracy but that's all. "Black guy -> gorilla" is just as likely as "yummyfajitas -> tank". Also, realistically, if we want to think about your black guy -> gorilla example, most likely if insufficient training data was the problem, then it was the number of gorillas that was too few. Google has vastly more black humans in their training set than gorillas - a quick google search suggests there are only 100,000 gorillas in the world, most of whom are probably unphotographed. As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be. Sure, algorithms are flawed. But this does not imply that bias must be present proportionally to ours. Insofar as we do good statistics, human bias is mitigated and can even go away. Science works. As a classical example, consider Morton's analysis of human skulls. Morton was a super racist guy trying to prove how white people are superior. But he did an unbiased analysis and accurately measured skull volume - filling a skull with buckshot is a pretty unbiased process. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001071 http://journals.plos.org/plosbiology/article?id=10.1371/jour... I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. What's controversial is that this is the idea of "original sin" but applied to science. And most of the proposed mechanisms are simply mathematically wrong. Also many of the examples cited by proponents of this original sin concept are also wrong.
    - muraiki 10y ago
      I appreciate your contribution to this discussion and will think about it further. Thanks.
- thaw13579 10y ago
  There was that twitter chatbot that learned some pretty awful things pretty quickly: http://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist http://www.theverge.com/2016/3/24/11297050/tay-microsoft-cha...

by muraiki 10y ago

yummyfajitas 10y ago

What evidence do you have that one specific classification error resulting in a tweetstorm was the result of bias at all? I've been mislabeled as a tank in at least one image processing demo (admittedly, this was well before the era of deep learning). Image classification errors happen. Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented among people building such algos, right? Consider the possibility that some classification problems are actually more difficult than others. As one example, darker objects are harder to distinguish than bright ones and there are very clear mathematical reasons for that. That's true even in fields where questions of "racism" don't make sense, e.g. in MRI (magnetic resonance imaging is non-optical, but contrast still matters): https://www.chrisstucchio.com/pubs/wf_segment.pdf https://www.chrisstucchio.com/pubs/wf_segment.pdf

muraiki 10y ago
I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning). As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be. Another example I heard recently from a conference was that of analyzing blighted homes in areas. The underlying data was generated by human inspectors who turned out to have given passes to violations in more affluent areas and fines in less affluent areas. The fines compounded the problem of blight as the residents could not afford to pay them and ended up causing a feedback loop of blight increasing in neighborhoods identified as blighted. I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. I apologize if I have only added confusion rather than a productive analysis.
- yummyfajitas 10y ago
  I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning). But that's not true. Inadequate training data will cause variance, not bias. This variance can have any direction. In a linear prediction scenario (i.e. any go/no go decision process), which exactly what Cathy O'Neil is criticizing, the effect could just as easily be "more loans for black people" as "less loans for black people". In a multidimensional predictor, it does directly decrease accuracy but that's all. "Black guy -> gorilla" is just as likely as "yummyfajitas -> tank". Also, realistically, if we want to think about your black guy -> gorilla example, most likely if insufficient training data was the problem, then it was the number of gorillas that was too few. Google has vastly more black humans in their training set than gorillas - a quick google search suggests there are only 100,000 gorillas in the world, most of whom are probably unphotographed. As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be. Sure, algorithms are flawed. But this does not imply that bias must be present proportionally to ours. Insofar as we do good statistics, human bias is mitigated and can even go away. Science works. As a classical example, consider Morton's analysis of human skulls. Morton was a super racist guy trying to prove how white people are superior. But he did an unbiased analysis and accurately measured skull volume - filling a skull with buckshot is a pretty unbiased process. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001071 http://journals.plos.org/plosbiology/article?id=10.1371/jour... I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. What's controversial is that this is the idea of "original sin" but applied to science. And most of the proposed mechanisms are simply mathematically wrong. Also many of the examples cited by proponents of this original sin concept are also wrong.
  - muraiki 10y ago
    I appreciate your contribution to this discussion and will think about it further. Thanks.
thaw13579 10y ago
There was that twitter chatbot that learned some pretty awful things pretty quickly: http://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist http://www.theverge.com/2016/3/24/11297050/tay-microsoft-cha...