6 ms·

Google Photos AI still can't label gorillas after racist errors

43 points by LinuxBender 3y ago

version_five 3y ago
I agree that blocking the feature is the smartest thing to do if people are going to get offended. It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem (I'd consider myself a computer vision expert). I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Identity politics aside, it would be an interesting study to try and break a man/gorilla classifier. Like take a picture of a man, say in a jungle setting and showing teeth or with a furry hat on, and see what the actual failure modes are. Regularly occurring misclassifications are a useful window into how a model operates.
- ChatGTP 3y ago
  [flagged]
  - noirbot 3y ago
    That would at least be an over-specific class though, as most colonialists are people. The issue here is that it's often not seeing people as people, and identifying them as animals. It would be more like if it ID'd white folks as cows or such.
    - rjh29 3y ago
      White folks = cows isn't a fair comparison, there is a strong racist history of comparing black people to monkeys/gorillas that increases the social harm.
      - noirbot 3y ago
        Sure. There's not a good comparison. But even if we're just talking about differences in training model issues, a portion of the underlying issue is that the classification is so wrong that it doesn't see some people as even being human. The fact that it's specifically gorillas is a level on top of that, but it would probably still be bad/a problem if it had the other common computer vision issue where it just doesn't see people with darker skin tones at all.
  - edgyquant 3y ago
    If it labeled me as a pig I’d understand it was a misclassification. As a colonizer I’d understand it was clearly someone injecting bias.
    - bannedbybros 3y ago
      [dead]
- jjevanoorschot 3y ago
  I'm on board with assuming that the intent of Google / the engineers who built the classifier is not racist. However the outcome — labelling a black person as a gorilla — certainly is racist. What makes you think otherwise?
  - syrrim 3y ago
    Presumably, the GP considers intent to be the only relevant factor in determining if something is racist.
    - rjh29 3y ago
      Intent WAS a factor here. There was no intent to consider anyone other than white people when the model was trained.
    - gadders 3y ago
      Leaving the current situation aside, it's an interesting philosophical point. In law you have the concept of "mens rea" https://en.wikipedia.org/wiki/Mens_rea https://en.wikipedia.org/wiki/Mens_rea
  - edgyquant 3y ago
    How is that racist? Because you’re projecting a racist comment onto it, a classifier? That does not make sense to me
    - deleted 3y ago
      [deleted]
    - giraffe_lady 3y ago
      Racism isn't an objective order existing in the universe separate from us. It's part of human experience and exists where humans experience it. Given the recent history of equating black people with non-human primates, and using that to deny them rights & full participation in society, making this error is going to be experienced as racist. It's not a matter of individual malice or taxonomic classification, but of history and social relations.
- im3w1l 3y ago
  My guess is that the model does work, but that verifying that it works is just way too expensive for a tiny benefit. Especially if the model is constantly evolving.
- throwawaysleep 3y ago
  > I wonder if they're over-reacting There doesn't seem to be any benefit to solving this problem, so how is it an overreaction given the negative press?
  - Larrikin 3y ago
    There's no benefit to you personally maybe? I'm sure there's a lot of black people who would like to be correctly identified instead of being labeled gorillas.
    - throwawaysleep 3y ago
      They aren't labelled gorillas. Google's API won't label gorillas.
    - brabel 3y ago
      They mean there's no benefit to correctly labelling gorillas so they keep that turned off completetly.
- jsnell 3y ago
  > At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem Why are you assuming that they can't get the classifier to work? There's no evidence to that effect in the article. > I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Why do you describe that as over-reacting? It seems like the appropriate level of reaction given there is no upside for getting it right and a massive downside for getting it wrong. It doesn't even need to be a top-down edict, it's just the way the incentives will work out for everyone involved. Let's say that you're an engineer working on that team, and find that there's a bunch of terms added to a blocklist a decade ago and still there. Are you going to just remove them, on the assumption that the classifications are correct? Of course not. How much validation work are you willing to do to convince yourself and others that the results are 100% correct? Is that really going to be the most interesting or impactful work you can do this quarter? Similarly you'd find that whoever needs to approve such a removal would have incentives very strongly biased toward not approving the removal of terms from a blocklist. If everything goes well, they get no credit. If something goes wrong, they get the blame. It's possible to push through that bias toward inaction, but it requires it to be a change that somebody really wants to make happen.
- prepend 3y ago
  Comically, the most potentially racist aspect (and I agree that having a bad classifier classify people as gorillas isn’t racist) is that Google hasn’t prioritized it enough in 8 years to fix. And this seems to fall into the sad, overly pervasive racism of people not caring vs the more explicit “a human calling someone a gorilla is racist” category. But also, it’s not like there’s many people who need to classify gorillas in photos so it likely doesn’t get brought up that much.
  - surgical_fire 3y ago
    > is that Google hasn’t prioritized it enough in 8 years to fix. I understand that there's nothing to fix. It doesn't label anything as gorillas anymore. Why waste effort in bringing it back, when another mislabeled photo will only serve for people to scream "racism" on social media? Better to not poke that particular beehive, I reckon.
    - prepend 3y ago
      The bug is in the classifier in that I would want a high accuracy in classifying. I doubt this is the only error it makes, just the most visible. So google isn’t prioritizing accurately classifying black people, just they hard coded to never classify gorillas. Presumably because their classifier is still wrong enough that it would think black people are not people. If they cared about this problem rather than just thinking it’s a “beehive” to be avoided rather than a real problem, then they would fix it. But they don’t actually care about accuracy in classifying people of some races, so they just avoid it. Not super racist, but definitely shows they don’t care about this type of bug in that the affected populations aren’t important to them. I understand not caring about gorillas, but I would expect they should be very interested in accurately classifying black people.
      - megaman821 3y ago
        Let's say the classified gets it right 999,999 times out of a million. It's about as fixed as it will ever be. Why would Google bring it back when there will be dozens of instances of them getting it wrong a year? It is mostly downside with very little upside. You can actually find things the Google probably thinks are gorillas and monkeys by searching for "zoo".
- mschuster91 3y ago
  > It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. To quote from the article: >> The company was criticized when a software developer, Jacky Alciné, found the image recognition system deployed in its Photos app in 2015 had mistakenly labelled a photo of him and his friend as gorillas. People who are black have every right to complain about Black people being classified as gorillas, particularly as that comparison is a very common act of discrimination - most infamously in soccer, where e.g. Vinícius Júnior is routinely subjected to monkey sounds and chants [1] and the President of the Spanish Agents association literally told him to "stop playing the monkey" as a goal celebration [2]. [1] https://www.cbssports.com/soccer/news/vinicius-junior-faces-monkey-chants-at-valencia-before-seeing-red-card-in-chaotic-clash-in-real-madrid-loss/ https://www.cbssports.com/soccer/news/vinicius-junior-faces-... [2] https://www.dailymail.co.uk/sport/sportsnews/article-11219075/Spanish-agent-tells-Vinicius-Jr-stop-playing-monkey-cocky-goal-celebration.html https://www.dailymail.co.uk/sport/sportsnews/article-1121907...
- jmull 3y ago
  > It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. It might be useful to understand why it's offensive... Likening black people to apes/gorillas/monkeys is a racist trope with a long history. It's part of casting black people as less than human, which has been used to justify their enslavement and denial of human rights. I think it's very likely the classifier was not constructed to be racist, but once you see that it is producing racist tropes, it is racist to continue to let it do so.
  - cookieperson 3y ago
    This. Once a bias is known it must be corrected. Allowing it to stand is basically saying "we don't care about black consumers, they can't use this feature, and we will continue to insult them on accident". Which it's no longer an accident, it's a known racially discriminative software feature... It's like hiring someone to do plumbing. A few customers invite them into their home and report that the new plumber said racist stuff. As a business owner you do what after this?
- Sparkyte 3y ago
  I would take being called a gorilla a compliment. They are jovial, endearing and typically well tempered. Humans on the otherhand I would almost consider an insult. Since we start massive wars, raise cultures and poke fun at each other in ways that are racist. Kinda bad when animals are better role models. But really it is history that is to blame. Gorillas are great. But if you called anyone a Lion or Lioness it would be perceived as a compliment. Gorillas don't deserve such connotation.
- photochemsyn 3y ago
  There are efforts to make more accurate classifiers (2021), which seems to come down to avoiding dataset biases, but for them to work, they essentially have to group humans into racial subsets (of which 'gorilla' and 'chimpanzee' might be a discrete entity, which is evolutionarily interesting at least, as humans/chimps/gorillas are more closely related than to each other relative any outgroup). https://www.mdpi.com/2227-7390/9/2/195 https://www.mdpi.com/2227-7390/9/2/195 > "Studies have shown that due to the distribution of ethnicity/race in training datasets, biometric algorithms suffer from “cross race effect”—their performance is better on subjects closer to the “country of origin” of the algorithm." However, these classifiers are also problematic as racial identification is a bit thorny politically. It's comparable to gender identification classifiers in the context of trans politics (as, regardless of personal identification, there are some classifier-accessible differences between the vast majority of female and male faces, although I suppose there's an androgenous middle ground). To go out on a limb a little, imagine an 'Aryan vs. Jewish' classifier - who wants to be associated with that? Lots of downsides for any commercial outfit, certainly, and few upsides.
- ausbah 3y ago
  I think it's idiotic and racist to dismiss people offense of being labeled as an animal with a long history of being used as derogatory racial insult
  - deleted 3y ago
    [deleted]
- aeturnum 3y ago
  > it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. Because of the entanglement of people who harbor individual racial animus and the systemic factors that lead to disparate outcomes for people of different races we are in a bad situation vis-a-vie language, but I think this isn't quite right. I agree there's no reason to believe the engineers involved had any prejudice (i.e. individualized animus against non-white people). It's the result of under-testing or corner cases or whatever. However, the system is definitely racist. It's racist in the sense that it has an adverse impact on people of different races (white people will not be mis-detected as gorillas) and its racist in the sense that there must have been racial imbalances in development (one struggles to imagine this being launched if all humans were being detected as gorillas for instance). This doesn't require any prejudice on the part of the people making it - it's just a fact that we can observe that it has disparate racial impacts and that choices in its development have led to a product that is unevenly useful to people of different races. It's also clear because facial recognition systems perform worse across all measures on people with darker skin (not just on this metric) - it's not like the accuracy rate is the same but the mistakes are different. Also, of course, yes - people will point out that different skin tones interact differently with photographic technology and algorithm designers are fighting an up hill battle. It's true! We have historically been good at solving the challenges of capturing images of light skinned people better than dark skinned ones. That's racist too! It's hard to not build a racist system when your tools already have racial bias. Again - pointing out the racial bias in this system is not an indictment of the people who built it. It's an indictment of the world it emerged out of and a bookmark to come back to when you see people using the outputs of the system.
- lisasays 3y ago
  It's idiotic to call it "racist", it's a shitty classifier This is sort of like telling residents of a historically black neighborhood which just so happens to have been exposed to drinking water with dangerously high levels of lead since time immemorial (despite decades of complaints) that it's just "shitty pipes", and that it's "idiotic" for anyone to think this state of affairs could possibly have something to do with racism. I agree that the matter is quite nuanced, and there was no racist intent. However it's far from "idiotic" to address these issues directly. If anything it seems a bit obtuse not to address them. Where there's smoke, after all, there's usually some form of fire.
  - prepend 3y ago
    I think the lead pipes example is different because both the neighborhood structure and the lead are results of the same intentional racism. The classifier is not the result of any racist action on behalf of the programmer. It’s more like blaming a hospital administrator when people of a discriminated race have bad outcomes at their hospital, when the outcome is the systemic rate of all the other hospitals. The hospital outcome isn’t racist, because the systemic issues that affect the outcomes of the hospital patients have nothing to do with the decisions of the hospital.
    - lisasays 3y ago
      Not necessarily -- recall that lead service lines were extremely widely used in the U.S. in all neighborhoods, black and white. So in fact it's more likely that he pipes being first put into the ground without any racist intent at all. It's the fact of this problem not being remediated in certain neighborhoods that cause people to ask, very legitimately -- how this came to pass.
spiderfarmer 3y ago
I read the entire wikipedia page on racism to see why people call a bug in an experimental product feature, based on bad classifiers and data, a form of racism. It's unfortunate. And I can see why it worries people. Because on the surface it looks like a systemic problem. But is it really?
- tornato7 3y ago
  Perhaps it is not racism that the bug exists, but failing to find, fix, or disable a feature that makes offensive classifications is racist.
throwaway29191 3y ago
This is a valuable lesson in what AI can do for society: act as a mirror to show us systemic biases that exist that people try to pretend don't exist. I think most here would understand systemic bias if reflected in a distributed system. Imagine a classifier where, through no explicit intent of the designer of the classifier, always said that some user login was malicious or some binary was malware or the like, even when it was not. And all those users or binaries had something in common that was apparent, and were all being misclassified. That would be systemic bias, and nobody would blink an eye about a call for it to be corrected. Here we see systemic racial bias, probably not because of individual racial animus on the part of some Google engineer, but because society has systemic racial bias and the classifier is just reflecting that because it was trained on public data. We should look at incidents like this differently: not as yet another chance to argue about what Google should or shouldn't do, but as a chance to see that a socially-neutral AI that does not have bias built into it is learning racial bias from society.
- pfannkuchen 3y ago
  > the classifier is just reflecting that because it was trained on public data Are you suggesting that the cause here is that there are a lot of racist images on the internet visually depicting black people as gorillas? I assume that is not the understanding of most people here who are questioning what makes this racist, because if that is indeed the cause then I agree it is obviously racist. Is it the cause though? I would be interested to see some evidence or rationale, because I don’t think I have ever seen an image like this outside of perhaps a handful of very old hand drawn propaganda images (which presumably would not affect the classifier much due to being overwhelmed by the vast number of images of real people and real gorillas).
- jl2718 3y ago
  There is a rigorous definition of bias, and that isn’t it. That is called ‘variance’. It doesn’t matter if there is structure to the variance. It doesn’t matter if the model is consistently wrong for one type of input. It is not bias unless it has been specifically and intentionally programmed to predict something other than the result from the maximum likelihood estimator given the data. A trickier type of bias is in the dataset itself, but once again, this would require a specific effort and intention to produce that bias by modifying the collection or filtering procedure. Nothing as much has been alleged, so you have provided no evidence of bias, ‘systemic’ or otherwise. Come up with a different word. Maybe “non-uniform variance”.
sashank_1509 3y ago
I think I could write a classifier that would accurately separate out gorillas and humans in a weekend with a dataset. But I guess as its pointed out here, there's no upside to getting it right and massive downside to getting it wrong
- lisasays 3y ago
  I suspect the challenge is not in getting that particular feature to work in isolation (which I agree seems trivial). But rather, in getting it to work in a way that integrates with the 10,000 other decision boundary problems that a large-scale AI system needs to be able to navigate. That is to say: not so challenging that they couldn't have got wrapped by now, if they cared enough to do so. But not necessarily trivial, either.
causi 3y ago
So is this a "racist data" thing or a "bad recognition" thing? Do pictures of white people get mislabeled as something else in ways pictures of other ethnicities don't?
we_never_see_it 3y ago
This proves my point. Google never had any lead in AI. All they have and will have is PR stunts.
bioemerl 3y ago
We need to be vigilant to no end about how important it is to be able to run AI systems ourselves and use the opportunity they provide without the oversight of HR or PR staff rooms.
ldehaan 3y ago
[dead]
yarg 3y ago
Both gorillas and black people have dark skin and reasonably flat features. The lack of contrast can make things hard the evaluate, especially if you're dealing with SDR content in a format that struggles to accurately encode darker colours. It's not racism, it's a lack of information to the evaluator and a failure of communication by the company. If they actually put out a statement regarding the technical issues around the problem, rather than trying to bury it in order to minimise the backlash, we could be having some far more reasonable and level headed discussions.
- pfannkuchen 3y ago
  Also a lot of the training set for “human” is on weird mutant albino humans who are over represented in images for historical reasons. It’s not offensive to look more like our relatives than the weird mutant albino alien looking creatures do, is it?
xnx 3y ago
Can't or won't
ausbah 3y ago
even if there's a perfectly reasonable & rational explanation for why classifiers like this make misclassifications like this, the fact that this sort of misclassification continues to be present is a widely used public facing model is racist. furthermore, attempts to dismiss the above as not being racist because "it's technical in nature" is also racist because it blatantly ignore the harm that comes from continuing to parrot a well know racial caricature

43 points by LinuxBender 3y ago

I agree that blocking the feature is the smartest thing to do if people are going to get offended. It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem (I'd consider myself a computer vision expert). I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Identity politics aside, it would be an interesting study to try and break a man/gorilla classifier. Like take a picture of a man, say in a jungle setting and showing teeth or with a furry hat on, and see what the actual failure modes are. Regularly occurring misclassifications are a useful window into how a model operates.

ChatGTP 3y ago
[flagged]
- noirbot 3y ago
  That would at least be an over-specific class though, as most colonialists are people. The issue here is that it's often not seeing people as people, and identifying them as animals. It would be more like if it ID'd white folks as cows or such.
  - rjh29 3y ago
    White folks = cows isn't a fair comparison, there is a strong racist history of comparing black people to monkeys/gorillas that increases the social harm.
    - noirbot 3y ago
      Sure. There's not a good comparison. But even if we're just talking about differences in training model issues, a portion of the underlying issue is that the classification is so wrong that it doesn't see some people as even being human. The fact that it's specifically gorillas is a level on top of that, but it would probably still be bad/a problem if it had the other common computer vision issue where it just doesn't see people with darker skin tones at all.
- edgyquant 3y ago
  If it labeled me as a pig I’d understand it was a misclassification. As a colonizer I’d understand it was clearly someone injecting bias.
  - bannedbybros 3y ago
    [dead]
jjevanoorschot 3y ago
I'm on board with assuming that the intent of Google / the engineers who built the classifier is not racist. However the outcome — labelling a black person as a gorilla — certainly is racist. What makes you think otherwise?
- syrrim 3y ago
  Presumably, the GP considers intent to be the only relevant factor in determining if something is racist.
  - rjh29 3y ago
    Intent WAS a factor here. There was no intent to consider anyone other than white people when the model was trained.
  - gadders 3y ago
    Leaving the current situation aside, it's an interesting philosophical point. In law you have the concept of "mens rea" https://en.wikipedia.org/wiki/Mens_rea https://en.wikipedia.org/wiki/Mens_rea
- edgyquant 3y ago
  How is that racist? Because you’re projecting a racist comment onto it, a classifier? That does not make sense to me
  - deleted 3y ago
    [deleted]
  - giraffe_lady 3y ago
    Racism isn't an objective order existing in the universe separate from us. It's part of human experience and exists where humans experience it. Given the recent history of equating black people with non-human primates, and using that to deny them rights & full participation in society, making this error is going to be experienced as racist. It's not a matter of individual malice or taxonomic classification, but of history and social relations.
im3w1l 3y ago
My guess is that the model does work, but that verifying that it works is just way too expensive for a tiny benefit. Especially if the model is constantly evolving.
throwawaysleep 3y ago
> I wonder if they're over-reacting There doesn't seem to be any benefit to solving this problem, so how is it an overreaction given the negative press?
- Larrikin 3y ago
  There's no benefit to you personally maybe? I'm sure there's a lot of black people who would like to be correctly identified instead of being labeled gorillas.
  - throwawaysleep 3y ago
    They aren't labelled gorillas. Google's API won't label gorillas.
  - brabel 3y ago
    They mean there's no benefit to correctly labelling gorillas so they keep that turned off completetly.
jsnell 3y ago
> At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem Why are you assuming that they can't get the classifier to work? There's no evidence to that effect in the article. > I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Why do you describe that as over-reacting? It seems like the appropriate level of reaction given there is no upside for getting it right and a massive downside for getting it wrong. It doesn't even need to be a top-down edict, it's just the way the incentives will work out for everyone involved. Let's say that you're an engineer working on that team, and find that there's a bunch of terms added to a blocklist a decade ago and still there. Are you going to just remove them, on the assumption that the classifications are correct? Of course not. How much validation work are you willing to do to convince yourself and others that the results are 100% correct? Is that really going to be the most interesting or impactful work you can do this quarter? Similarly you'd find that whoever needs to approve such a removal would have incentives very strongly biased toward not approving the removal of terms from a blocklist. If everything goes well, they get no credit. If something goes wrong, they get the blame. It's possible to push through that bias toward inaction, but it requires it to be a change that somebody really wants to make happen.
prepend 3y ago
Comically, the most potentially racist aspect (and I agree that having a bad classifier classify people as gorillas isn’t racist) is that Google hasn’t prioritized it enough in 8 years to fix. And this seems to fall into the sad, overly pervasive racism of people not caring vs the more explicit “a human calling someone a gorilla is racist” category. But also, it’s not like there’s many people who need to classify gorillas in photos so it likely doesn’t get brought up that much.
- surgical_fire 3y ago
  > is that Google hasn’t prioritized it enough in 8 years to fix. I understand that there's nothing to fix. It doesn't label anything as gorillas anymore. Why waste effort in bringing it back, when another mislabeled photo will only serve for people to scream "racism" on social media? Better to not poke that particular beehive, I reckon.
  - prepend 3y ago
    The bug is in the classifier in that I would want a high accuracy in classifying. I doubt this is the only error it makes, just the most visible. So google isn’t prioritizing accurately classifying black people, just they hard coded to never classify gorillas. Presumably because their classifier is still wrong enough that it would think black people are not people. If they cared about this problem rather than just thinking it’s a “beehive” to be avoided rather than a real problem, then they would fix it. But they don’t actually care about accuracy in classifying people of some races, so they just avoid it. Not super racist, but definitely shows they don’t care about this type of bug in that the affected populations aren’t important to them. I understand not caring about gorillas, but I would expect they should be very interested in accurately classifying black people.
    - megaman821 3y ago
      Let's say the classified gets it right 999,999 times out of a million. It's about as fixed as it will ever be. Why would Google bring it back when there will be dozens of instances of them getting it wrong a year? It is mostly downside with very little upside. You can actually find things the Google probably thinks are gorillas and monkeys by searching for "zoo".
mschuster91 3y ago
> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. To quote from the article: >> The company was criticized when a software developer, Jacky Alciné, found the image recognition system deployed in its Photos app in 2015 had mistakenly labelled a photo of him and his friend as gorillas. People who are black have every right to complain about Black people being classified as gorillas, particularly as that comparison is a very common act of discrimination - most infamously in soccer, where e.g. Vinícius Júnior is routinely subjected to monkey sounds and chants [1] and the President of the Spanish Agents association literally told him to "stop playing the monkey" as a goal celebration [2]. [1] https://www.cbssports.com/soccer/news/vinicius-junior-faces-monkey-chants-at-valencia-before-seeing-red-card-in-chaotic-clash-in-real-madrid-loss/ https://www.cbssports.com/soccer/news/vinicius-junior-faces-... [2] https://www.dailymail.co.uk/sport/sportsnews/article-11219075/Spanish-agent-tells-Vinicius-Jr-stop-playing-monkey-cocky-goal-celebration.html https://www.dailymail.co.uk/sport/sportsnews/article-1121907...
jmull 3y ago
> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. It might be useful to understand why it's offensive... Likening black people to apes/gorillas/monkeys is a racist trope with a long history. It's part of casting black people as less than human, which has been used to justify their enslavement and denial of human rights. I think it's very likely the classifier was not constructed to be racist, but once you see that it is producing racist tropes, it is racist to continue to let it do so.
- cookieperson 3y ago
  This. Once a bias is known it must be corrected. Allowing it to stand is basically saying "we don't care about black consumers, they can't use this feature, and we will continue to insult them on accident". Which it's no longer an accident, it's a known racially discriminative software feature... It's like hiring someone to do plumbing. A few customers invite them into their home and report that the new plumber said racist stuff. As a business owner you do what after this?
Sparkyte 3y ago
I would take being called a gorilla a compliment. They are jovial, endearing and typically well tempered. Humans on the otherhand I would almost consider an insult. Since we start massive wars, raise cultures and poke fun at each other in ways that are racist. Kinda bad when animals are better role models. But really it is history that is to blame. Gorillas are great. But if you called anyone a Lion or Lioness it would be perceived as a compliment. Gorillas don't deserve such connotation.
photochemsyn 3y ago
There are efforts to make more accurate classifiers (2021), which seems to come down to avoiding dataset biases, but for them to work, they essentially have to group humans into racial subsets (of which 'gorilla' and 'chimpanzee' might be a discrete entity, which is evolutionarily interesting at least, as humans/chimps/gorillas are more closely related than to each other relative any outgroup). https://www.mdpi.com/2227-7390/9/2/195 https://www.mdpi.com/2227-7390/9/2/195 > "Studies have shown that due to the distribution of ethnicity/race in training datasets, biometric algorithms suffer from “cross race effect”—their performance is better on subjects closer to the “country of origin” of the algorithm." However, these classifiers are also problematic as racial identification is a bit thorny politically. It's comparable to gender identification classifiers in the context of trans politics (as, regardless of personal identification, there are some classifier-accessible differences between the vast majority of female and male faces, although I suppose there's an androgenous middle ground). To go out on a limb a little, imagine an 'Aryan vs. Jewish' classifier - who wants to be associated with that? Lots of downsides for any commercial outfit, certainly, and few upsides.
ausbah 3y ago
I think it's idiotic and racist to dismiss people offense of being labeled as an animal with a long history of being used as derogatory racial insult
- deleted 3y ago
  [deleted]
aeturnum 3y ago
> it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. Because of the entanglement of people who harbor individual racial animus and the systemic factors that lead to disparate outcomes for people of different races we are in a bad situation vis-a-vie language, but I think this isn't quite right. I agree there's no reason to believe the engineers involved had any prejudice (i.e. individualized animus against non-white people). It's the result of under-testing or corner cases or whatever. However, the system is definitely racist. It's racist in the sense that it has an adverse impact on people of different races (white people will not be mis-detected as gorillas) and its racist in the sense that there must have been racial imbalances in development (one struggles to imagine this being launched if all humans were being detected as gorillas for instance). This doesn't require any prejudice on the part of the people making it - it's just a fact that we can observe that it has disparate racial impacts and that choices in its development have led to a product that is unevenly useful to people of different races. It's also clear because facial recognition systems perform worse across all measures on people with darker skin (not just on this metric) - it's not like the accuracy rate is the same but the mistakes are different. Also, of course, yes - people will point out that different skin tones interact differently with photographic technology and algorithm designers are fighting an up hill battle. It's true! We have historically been good at solving the challenges of capturing images of light skinned people better than dark skinned ones. That's racist too! It's hard to not build a racist system when your tools already have racial bias. Again - pointing out the racial bias in this system is not an indictment of the people who built it. It's an indictment of the world it emerged out of and a bookmark to come back to when you see people using the outputs of the system.
lisasays 3y ago
It's idiotic to call it "racist", it's a shitty classifier This is sort of like telling residents of a historically black neighborhood which just so happens to have been exposed to drinking water with dangerously high levels of lead since time immemorial (despite decades of complaints) that it's just "shitty pipes", and that it's "idiotic" for anyone to think this state of affairs could possibly have something to do with racism. I agree that the matter is quite nuanced, and there was no racist intent. However it's far from "idiotic" to address these issues directly. If anything it seems a bit obtuse not to address them. Where there's smoke, after all, there's usually some form of fire.
- prepend 3y ago
  I think the lead pipes example is different because both the neighborhood structure and the lead are results of the same intentional racism. The classifier is not the result of any racist action on behalf of the programmer. It’s more like blaming a hospital administrator when people of a discriminated race have bad outcomes at their hospital, when the outcome is the systemic rate of all the other hospitals. The hospital outcome isn’t racist, because the systemic issues that affect the outcomes of the hospital patients have nothing to do with the decisions of the hospital.
  - lisasays 3y ago
    Not necessarily -- recall that lead service lines were extremely widely used in the U.S. in all neighborhoods, black and white. So in fact it's more likely that he pipes being first put into the ground without any racist intent at all. It's the fact of this problem not being remediated in certain neighborhoods that cause people to ask, very legitimately -- how this came to pass.

spiderfarmer 3y ago

I read the entire wikipedia page on racism to see why people call a bug in an experimental product feature, based on bad classifiers and data, a form of racism. It's unfortunate. And I can see why it worries people. Because on the surface it looks like a systemic problem. But is it really?

tornato7 3y ago
Perhaps it is not racism that the bug exists, but failing to find, fix, or disable a feature that makes offensive classifications is racist.

throwaway29191 3y ago

This is a valuable lesson in what AI can do for society: act as a mirror to show us systemic biases that exist that people try to pretend don't exist. I think most here would understand systemic bias if reflected in a distributed system. Imagine a classifier where, through no explicit intent of the designer of the classifier, always said that some user login was malicious or some binary was malware or the like, even when it was not. And all those users or binaries had something in common that was apparent, and were all being misclassified. That would be systemic bias, and nobody would blink an eye about a call for it to be corrected. Here we see systemic racial bias, probably not because of individual racial animus on the part of some Google engineer, but because society has systemic racial bias and the classifier is just reflecting that because it was trained on public data. We should look at incidents like this differently: not as yet another chance to argue about what Google should or shouldn't do, but as a chance to see that a socially-neutral AI that does not have bias built into it is learning racial bias from society.

pfannkuchen 3y ago
> the classifier is just reflecting that because it was trained on public data Are you suggesting that the cause here is that there are a lot of racist images on the internet visually depicting black people as gorillas? I assume that is not the understanding of most people here who are questioning what makes this racist, because if that is indeed the cause then I agree it is obviously racist. Is it the cause though? I would be interested to see some evidence or rationale, because I don’t think I have ever seen an image like this outside of perhaps a handful of very old hand drawn propaganda images (which presumably would not affect the classifier much due to being overwhelmed by the vast number of images of real people and real gorillas).
jl2718 3y ago
There is a rigorous definition of bias, and that isn’t it. That is called ‘variance’. It doesn’t matter if there is structure to the variance. It doesn’t matter if the model is consistently wrong for one type of input. It is not bias unless it has been specifically and intentionally programmed to predict something other than the result from the maximum likelihood estimator given the data. A trickier type of bias is in the dataset itself, but once again, this would require a specific effort and intention to produce that bias by modifying the collection or filtering procedure. Nothing as much has been alleged, so you have provided no evidence of bias, ‘systemic’ or otherwise. Come up with a different word. Maybe “non-uniform variance”.

sashank_1509 3y ago

I think I could write a classifier that would accurately separate out gorillas and humans in a weekend with a dataset. But I guess as its pointed out here, there's no upside to getting it right and massive downside to getting it wrong

lisasays 3y ago
I suspect the challenge is not in getting that particular feature to work in isolation (which I agree seems trivial). But rather, in getting it to work in a way that integrates with the 10,000 other decision boundary problems that a large-scale AI system needs to be able to navigate. That is to say: not so challenging that they couldn't have got wrapped by now, if they cared enough to do so. But not necessarily trivial, either.

causi 3y ago

So is this a "racist data" thing or a "bad recognition" thing? Do pictures of white people get mislabeled as something else in ways pictures of other ethnicities don't?

we_never_see_it 3y ago

This proves my point. Google never had any lead in AI. All they have and will have is PR stunts.

bioemerl 3y ago

We need to be vigilant to no end about how important it is to be able to run AI systems ourselves and use the opportunity they provide without the oversight of HR or PR staff rooms.

ldehaan 3y ago

[dead]

yarg 3y ago

Both gorillas and black people have dark skin and reasonably flat features. The lack of contrast can make things hard the evaluate, especially if you're dealing with SDR content in a format that struggles to accurately encode darker colours. It's not racism, it's a lack of information to the evaluator and a failure of communication by the company. If they actually put out a statement regarding the technical issues around the problem, rather than trying to bury it in order to minimise the backlash, we could be having some far more reasonable and level headed discussions.

pfannkuchen 3y ago
Also a lot of the training set for “human” is on weird mutant albino humans who are over represented in images for historical reasons. It’s not offensive to look more like our relatives than the weird mutant albino alien looking creatures do, is it?

xnx 3y ago

Can't or won't

ausbah 3y ago

even if there's a perfectly reasonable & rational explanation for why classifiers like this make misclassifications like this, the fact that this sort of misclassification continues to be present is a widely used public facing model is racist. furthermore, attempts to dismiss the above as not being racist because "it's technical in nature" is also racist because it blatantly ignore the harm that comes from continuing to parrot a well know racial caricature