6 ms·

I agree that blocking the feature is the smartest thing to do if people are going to get offended. It's idiotic to call it "racist", it's a shitty classifier, a

I agree that blocking the feature is the smartest thing to do if people are going to get offended. It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem (I'd consider myself a computer vision expert). I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Identity politics aside, it would be an interesting study to try and break a man/gorilla classifier. Like take a picture of a man, say in a jungle setting and showing teeth or with a furry hat on, and see what the actual failure modes are. Regularly occurring misclassifications are a useful window into how a model operates.

ChatGTP 3y ago
[flagged]
- noirbot 3y ago
  That would at least be an over-specific class though, as most colonialists are people. The issue here is that it's often not seeing people as people, and identifying them as animals. It would be more like if it ID'd white folks as cows or such.
  - rjh29 3y ago
    White folks = cows isn't a fair comparison, there is a strong racist history of comparing black people to monkeys/gorillas that increases the social harm.
    - noirbot 3y ago
      Sure. There's not a good comparison. But even if we're just talking about differences in training model issues, a portion of the underlying issue is that the classification is so wrong that it doesn't see some people as even being human. The fact that it's specifically gorillas is a level on top of that, but it would probably still be bad/a problem if it had the other common computer vision issue where it just doesn't see people with darker skin tones at all.
- edgyquant 3y ago
  If it labeled me as a pig I’d understand it was a misclassification. As a colonizer I’d understand it was clearly someone injecting bias.
  - bannedbybros 3y ago
    [dead]
jjevanoorschot 3y ago
I'm on board with assuming that the intent of Google / the engineers who built the classifier is not racist. However the outcome — labelling a black person as a gorilla — certainly is racist. What makes you think otherwise?
- syrrim 3y ago
  Presumably, the GP considers intent to be the only relevant factor in determining if something is racist.
  - rjh29 3y ago
    Intent WAS a factor here. There was no intent to consider anyone other than white people when the model was trained.
  - gadders 3y ago
    Leaving the current situation aside, it's an interesting philosophical point. In law you have the concept of "mens rea" https://en.wikipedia.org/wiki/Mens_rea https://en.wikipedia.org/wiki/Mens_rea
- edgyquant 3y ago
  How is that racist? Because you’re projecting a racist comment onto it, a classifier? That does not make sense to me
  - deleted 3y ago
    [deleted]
  - giraffe_lady 3y ago
    Racism isn't an objective order existing in the universe separate from us. It's part of human experience and exists where humans experience it. Given the recent history of equating black people with non-human primates, and using that to deny them rights & full participation in society, making this error is going to be experienced as racist. It's not a matter of individual malice or taxonomic classification, but of history and social relations.
im3w1l 3y ago
My guess is that the model does work, but that verifying that it works is just way too expensive for a tiny benefit. Especially if the model is constantly evolving.
throwawaysleep 3y ago
> I wonder if they're over-reacting There doesn't seem to be any benefit to solving this problem, so how is it an overreaction given the negative press?
- Larrikin 3y ago
  There's no benefit to you personally maybe? I'm sure there's a lot of black people who would like to be correctly identified instead of being labeled gorillas.
  - throwawaysleep 3y ago
    They aren't labelled gorillas. Google's API won't label gorillas.
  - brabel 3y ago
    They mean there's no benefit to correctly labelling gorillas so they keep that turned off completetly.
jsnell 3y ago
> At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem Why are you assuming that they can't get the classifier to work? There's no evidence to that effect in the article. > I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Why do you describe that as over-reacting? It seems like the appropriate level of reaction given there is no upside for getting it right and a massive downside for getting it wrong. It doesn't even need to be a top-down edict, it's just the way the incentives will work out for everyone involved. Let's say that you're an engineer working on that team, and find that there's a bunch of terms added to a blocklist a decade ago and still there. Are you going to just remove them, on the assumption that the classifications are correct? Of course not. How much validation work are you willing to do to convince yourself and others that the results are 100% correct? Is that really going to be the most interesting or impactful work you can do this quarter? Similarly you'd find that whoever needs to approve such a removal would have incentives very strongly biased toward not approving the removal of terms from a blocklist. If everything goes well, they get no credit. If something goes wrong, they get the blame. It's possible to push through that bias toward inaction, but it requires it to be a change that somebody really wants to make happen.
prepend 3y ago
Comically, the most potentially racist aspect (and I agree that having a bad classifier classify people as gorillas isn’t racist) is that Google hasn’t prioritized it enough in 8 years to fix. And this seems to fall into the sad, overly pervasive racism of people not caring vs the more explicit “a human calling someone a gorilla is racist” category. But also, it’s not like there’s many people who need to classify gorillas in photos so it likely doesn’t get brought up that much.
- surgical_fire 3y ago
  > is that Google hasn’t prioritized it enough in 8 years to fix. I understand that there's nothing to fix. It doesn't label anything as gorillas anymore. Why waste effort in bringing it back, when another mislabeled photo will only serve for people to scream "racism" on social media? Better to not poke that particular beehive, I reckon.
  - prepend 3y ago
    The bug is in the classifier in that I would want a high accuracy in classifying. I doubt this is the only error it makes, just the most visible. So google isn’t prioritizing accurately classifying black people, just they hard coded to never classify gorillas. Presumably because their classifier is still wrong enough that it would think black people are not people. If they cared about this problem rather than just thinking it’s a “beehive” to be avoided rather than a real problem, then they would fix it. But they don’t actually care about accuracy in classifying people of some races, so they just avoid it. Not super racist, but definitely shows they don’t care about this type of bug in that the affected populations aren’t important to them. I understand not caring about gorillas, but I would expect they should be very interested in accurately classifying black people.
    - megaman821 3y ago
      Let's say the classified gets it right 999,999 times out of a million. It's about as fixed as it will ever be. Why would Google bring it back when there will be dozens of instances of them getting it wrong a year? It is mostly downside with very little upside. You can actually find things the Google probably thinks are gorillas and monkeys by searching for "zoo".
mschuster91 3y ago
> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. To quote from the article: >> The company was criticized when a software developer, Jacky Alciné, found the image recognition system deployed in its Photos app in 2015 had mistakenly labelled a photo of him and his friend as gorillas. People who are black have every right to complain about Black people being classified as gorillas, particularly as that comparison is a very common act of discrimination - most infamously in soccer, where e.g. Vinícius Júnior is routinely subjected to monkey sounds and chants [1] and the President of the Spanish Agents association literally told him to "stop playing the monkey" as a goal celebration [2]. [1] https://www.cbssports.com/soccer/news/vinicius-junior-faces-monkey-chants-at-valencia-before-seeing-red-card-in-chaotic-clash-in-real-madrid-loss/ https://www.cbssports.com/soccer/news/vinicius-junior-faces-... [2] https://www.dailymail.co.uk/sport/sportsnews/article-11219075/Spanish-agent-tells-Vinicius-Jr-stop-playing-monkey-cocky-goal-celebration.html https://www.dailymail.co.uk/sport/sportsnews/article-1121907...
jmull 3y ago
> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. It might be useful to understand why it's offensive... Likening black people to apes/gorillas/monkeys is a racist trope with a long history. It's part of casting black people as less than human, which has been used to justify their enslavement and denial of human rights. I think it's very likely the classifier was not constructed to be racist, but once you see that it is producing racist tropes, it is racist to continue to let it do so.
- cookieperson 3y ago
  This. Once a bias is known it must be corrected. Allowing it to stand is basically saying "we don't care about black consumers, they can't use this feature, and we will continue to insult them on accident". Which it's no longer an accident, it's a known racially discriminative software feature... It's like hiring someone to do plumbing. A few customers invite them into their home and report that the new plumber said racist stuff. As a business owner you do what after this?
Sparkyte 3y ago
I would take being called a gorilla a compliment. They are jovial, endearing and typically well tempered. Humans on the otherhand I would almost consider an insult. Since we start massive wars, raise cultures and poke fun at each other in ways that are racist. Kinda bad when animals are better role models. But really it is history that is to blame. Gorillas are great. But if you called anyone a Lion or Lioness it would be perceived as a compliment. Gorillas don't deserve such connotation.
photochemsyn 3y ago
There are efforts to make more accurate classifiers (2021), which seems to come down to avoiding dataset biases, but for them to work, they essentially have to group humans into racial subsets (of which 'gorilla' and 'chimpanzee' might be a discrete entity, which is evolutionarily interesting at least, as humans/chimps/gorillas are more closely related than to each other relative any outgroup). https://www.mdpi.com/2227-7390/9/2/195 https://www.mdpi.com/2227-7390/9/2/195 > "Studies have shown that due to the distribution of ethnicity/race in training datasets, biometric algorithms suffer from “cross race effect”—their performance is better on subjects closer to the “country of origin” of the algorithm." However, these classifiers are also problematic as racial identification is a bit thorny politically. It's comparable to gender identification classifiers in the context of trans politics (as, regardless of personal identification, there are some classifier-accessible differences between the vast majority of female and male faces, although I suppose there's an androgenous middle ground). To go out on a limb a little, imagine an 'Aryan vs. Jewish' classifier - who wants to be associated with that? Lots of downsides for any commercial outfit, certainly, and few upsides.
ausbah 3y ago
I think it's idiotic and racist to dismiss people offense of being labeled as an animal with a long history of being used as derogatory racial insult
- deleted 3y ago
  [deleted]
aeturnum 3y ago
> it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. Because of the entanglement of people who harbor individual racial animus and the systemic factors that lead to disparate outcomes for people of different races we are in a bad situation vis-a-vie language, but I think this isn't quite right. I agree there's no reason to believe the engineers involved had any prejudice (i.e. individualized animus against non-white people). It's the result of under-testing or corner cases or whatever. However, the system is definitely racist. It's racist in the sense that it has an adverse impact on people of different races (white people will not be mis-detected as gorillas) and its racist in the sense that there must have been racial imbalances in development (one struggles to imagine this being launched if all humans were being detected as gorillas for instance). This doesn't require any prejudice on the part of the people making it - it's just a fact that we can observe that it has disparate racial impacts and that choices in its development have led to a product that is unevenly useful to people of different races. It's also clear because facial recognition systems perform worse across all measures on people with darker skin (not just on this metric) - it's not like the accuracy rate is the same but the mistakes are different. Also, of course, yes - people will point out that different skin tones interact differently with photographic technology and algorithm designers are fighting an up hill battle. It's true! We have historically been good at solving the challenges of capturing images of light skinned people better than dark skinned ones. That's racist too! It's hard to not build a racist system when your tools already have racial bias. Again - pointing out the racial bias in this system is not an indictment of the people who built it. It's an indictment of the world it emerged out of and a bookmark to come back to when you see people using the outputs of the system.
lisasays 3y ago
It's idiotic to call it "racist", it's a shitty classifier This is sort of like telling residents of a historically black neighborhood which just so happens to have been exposed to drinking water with dangerously high levels of lead since time immemorial (despite decades of complaints) that it's just "shitty pipes", and that it's "idiotic" for anyone to think this state of affairs could possibly have something to do with racism. I agree that the matter is quite nuanced, and there was no racist intent. However it's far from "idiotic" to address these issues directly. If anything it seems a bit obtuse not to address them. Where there's smoke, after all, there's usually some form of fire.
- prepend 3y ago
  I think the lead pipes example is different because both the neighborhood structure and the lead are results of the same intentional racism. The classifier is not the result of any racist action on behalf of the programmer. It’s more like blaming a hospital administrator when people of a discriminated race have bad outcomes at their hospital, when the outcome is the systemic rate of all the other hospitals. The hospital outcome isn’t racist, because the systemic issues that affect the outcomes of the hospital patients have nothing to do with the decisions of the hospital.
  - lisasays 3y ago
    Not necessarily -- recall that lead service lines were extremely widely used in the U.S. in all neighborhoods, black and white. So in fact it's more likely that he pipes being first put into the ground without any racist intent at all. It's the fact of this problem not being remediated in certain neighborhoods that cause people to ask, very legitimately -- how this came to pass.

ChatGTP 3y ago

[flagged]

noirbot 3y ago
That would at least be an over-specific class though, as most colonialists are people. The issue here is that it's often not seeing people as people, and identifying them as animals. It would be more like if it ID'd white folks as cows or such.
- rjh29 3y ago
  White folks = cows isn't a fair comparison, there is a strong racist history of comparing black people to monkeys/gorillas that increases the social harm.
  - noirbot 3y ago
    Sure. There's not a good comparison. But even if we're just talking about differences in training model issues, a portion of the underlying issue is that the classification is so wrong that it doesn't see some people as even being human. The fact that it's specifically gorillas is a level on top of that, but it would probably still be bad/a problem if it had the other common computer vision issue where it just doesn't see people with darker skin tones at all.
edgyquant 3y ago
If it labeled me as a pig I’d understand it was a misclassification. As a colonizer I’d understand it was clearly someone injecting bias.
- bannedbybros 3y ago
  [dead]

jjevanoorschot 3y ago

I'm on board with assuming that the intent of Google / the engineers who built the classifier is not racist. However the outcome — labelling a black person as a gorilla — certainly is racist. What makes you think otherwise?

syrrim 3y ago
Presumably, the GP considers intent to be the only relevant factor in determining if something is racist.
- rjh29 3y ago
  Intent WAS a factor here. There was no intent to consider anyone other than white people when the model was trained.
- gadders 3y ago
  Leaving the current situation aside, it's an interesting philosophical point. In law you have the concept of "mens rea" https://en.wikipedia.org/wiki/Mens_rea https://en.wikipedia.org/wiki/Mens_rea
edgyquant 3y ago
How is that racist? Because you’re projecting a racist comment onto it, a classifier? That does not make sense to me
- deleted 3y ago
  [deleted]
- giraffe_lady 3y ago
  Racism isn't an objective order existing in the universe separate from us. It's part of human experience and exists where humans experience it. Given the recent history of equating black people with non-human primates, and using that to deny them rights & full participation in society, making this error is going to be experienced as racist. It's not a matter of individual malice or taxonomic classification, but of history and social relations.

im3w1l 3y ago

My guess is that the model does work, but that verifying that it works is just way too expensive for a tiny benefit. Especially if the model is constantly evolving.

throwawaysleep 3y ago

> I wonder if they're over-reacting There doesn't seem to be any benefit to solving this problem, so how is it an overreaction given the negative press?

Larrikin 3y ago
There's no benefit to you personally maybe? I'm sure there's a lot of black people who would like to be correctly identified instead of being labeled gorillas.
- throwawaysleep 3y ago
  They aren't labelled gorillas. Google's API won't label gorillas.
- brabel 3y ago
  They mean there's no benefit to correctly labelling gorillas so they keep that turned off completetly.

jsnell 3y ago

> At the same time, I'm surprised they can't get this classifier to work. It doesn't seem like a very challenging problem Why are you assuming that they can't get the classifier to work? There's no evidence to that effect in the article. > I wonder if they're over-reacting and just deciding to zero any residual risk by not allowing that classification anymore. Why do you describe that as over-reacting? It seems like the appropriate level of reaction given there is no upside for getting it right and a massive downside for getting it wrong. It doesn't even need to be a top-down edict, it's just the way the incentives will work out for everyone involved. Let's say that you're an engineer working on that team, and find that there's a bunch of terms added to a blocklist a decade ago and still there. Are you going to just remove them, on the assumption that the classifications are correct? Of course not. How much validation work are you willing to do to convince yourself and others that the results are 100% correct? Is that really going to be the most interesting or impactful work you can do this quarter? Similarly you'd find that whoever needs to approve such a removal would have incentives very strongly biased toward not approving the removal of terms from a blocklist. If everything goes well, they get no credit. If something goes wrong, they get the blame. It's possible to push through that bias toward inaction, but it requires it to be a change that somebody really wants to make happen.

prepend 3y ago

Comically, the most potentially racist aspect (and I agree that having a bad classifier classify people as gorillas isn’t racist) is that Google hasn’t prioritized it enough in 8 years to fix. And this seems to fall into the sad, overly pervasive racism of people not caring vs the more explicit “a human calling someone a gorilla is racist” category. But also, it’s not like there’s many people who need to classify gorillas in photos so it likely doesn’t get brought up that much.

surgical_fire 3y ago
> is that Google hasn’t prioritized it enough in 8 years to fix. I understand that there's nothing to fix. It doesn't label anything as gorillas anymore. Why waste effort in bringing it back, when another mislabeled photo will only serve for people to scream "racism" on social media? Better to not poke that particular beehive, I reckon.
- prepend 3y ago
  The bug is in the classifier in that I would want a high accuracy in classifying. I doubt this is the only error it makes, just the most visible. So google isn’t prioritizing accurately classifying black people, just they hard coded to never classify gorillas. Presumably because their classifier is still wrong enough that it would think black people are not people. If they cared about this problem rather than just thinking it’s a “beehive” to be avoided rather than a real problem, then they would fix it. But they don’t actually care about accuracy in classifying people of some races, so they just avoid it. Not super racist, but definitely shows they don’t care about this type of bug in that the affected populations aren’t important to them. I understand not caring about gorillas, but I would expect they should be very interested in accurately classifying black people.
  - megaman821 3y ago
    Let's say the classified gets it right 999,999 times out of a million. It's about as fixed as it will ever be. Why would Google bring it back when there will be dozens of instances of them getting it wrong a year? It is mostly downside with very little upside. You can actually find things the Google probably thinks are gorillas and monkeys by searching for "zoo".

mschuster91 3y ago

> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. To quote from the article: >> The company was criticized when a software developer, Jacky Alciné, found the image recognition system deployed in its Photos app in 2015 had mistakenly labelled a photo of him and his friend as gorillas. People who are black have every right to complain about Black people being classified as gorillas, particularly as that comparison is a very common act of discrimination - most infamously in soccer, where e.g. Vinícius Júnior is routinely subjected to monkey sounds and chants [1] and the President of the Spanish Agents association literally told him to "stop playing the monkey" as a goal celebration [2]. [1] https://www.cbssports.com/soccer/news/vinicius-junior-faces-monkey-chants-at-valencia-before-seeing-red-card-in-chaotic-clash-in-real-madrid-loss/ https://www.cbssports.com/soccer/news/vinicius-junior-faces-... [2] https://www.dailymail.co.uk/sport/sportsnews/article-11219075/Spanish-agent-tells-Vinicius-Jr-stop-playing-monkey-cocky-goal-celebration.html https://www.dailymail.co.uk/sport/sportsnews/article-1121907...

jmull 3y ago

> It's idiotic to call it "racist", it's a shitty classifier, and it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. It might be useful to understand why it's offensive... Likening black people to apes/gorillas/monkeys is a racist trope with a long history. It's part of casting black people as less than human, which has been used to justify their enslavement and denial of human rights. I think it's very likely the classifier was not constructed to be racist, but once you see that it is producing racist tropes, it is racist to continue to let it do so.

cookieperson 3y ago
This. Once a bias is known it must be corrected. Allowing it to stand is basically saying "we don't care about black consumers, they can't use this feature, and we will continue to insult them on accident". Which it's no longer an accident, it's a known racially discriminative software feature... It's like hiring someone to do plumbing. A few customers invite them into their home and report that the new plumber said racist stuff. As a business owner you do what after this?

Sparkyte 3y ago

I would take being called a gorilla a compliment. They are jovial, endearing and typically well tempered. Humans on the otherhand I would almost consider an insult. Since we start massive wars, raise cultures and poke fun at each other in ways that are racist. Kinda bad when animals are better role models. But really it is history that is to blame. Gorillas are great. But if you called anyone a Lion or Lioness it would be perceived as a compliment. Gorillas don't deserve such connotation.

photochemsyn 3y ago

There are efforts to make more accurate classifiers (2021), which seems to come down to avoiding dataset biases, but for them to work, they essentially have to group humans into racial subsets (of which 'gorilla' and 'chimpanzee' might be a discrete entity, which is evolutionarily interesting at least, as humans/chimps/gorillas are more closely related than to each other relative any outgroup). https://www.mdpi.com/2227-7390/9/2/195 https://www.mdpi.com/2227-7390/9/2/195 > "Studies have shown that due to the distribution of ethnicity/race in training datasets, biometric algorithms suffer from “cross race effect”—their performance is better on subjects closer to the “country of origin” of the algorithm." However, these classifiers are also problematic as racial identification is a bit thorny politically. It's comparable to gender identification classifiers in the context of trans politics (as, regardless of personal identification, there are some classifier-accessible differences between the vast majority of female and male faces, although I suppose there's an androgenous middle ground). To go out on a limb a little, imagine an 'Aryan vs. Jewish' classifier - who wants to be associated with that? Lots of downsides for any commercial outfit, certainly, and few upsides.

ausbah 3y ago

I think it's idiotic and racist to dismiss people offense of being labeled as an animal with a long history of being used as derogatory racial insult

deleted 3y ago
[deleted]

aeturnum 3y ago

> it's incredibly irresponsible (and probably racist) as well as dismissive of actual racism to pretend that's racist. Because of the entanglement of people who harbor individual racial animus and the systemic factors that lead to disparate outcomes for people of different races we are in a bad situation vis-a-vie language, but I think this isn't quite right. I agree there's no reason to believe the engineers involved had any prejudice (i.e. individualized animus against non-white people). It's the result of under-testing or corner cases or whatever. However, the system is definitely racist. It's racist in the sense that it has an adverse impact on people of different races (white people will not be mis-detected as gorillas) and its racist in the sense that there must have been racial imbalances in development (one struggles to imagine this being launched if all humans were being detected as gorillas for instance). This doesn't require any prejudice on the part of the people making it - it's just a fact that we can observe that it has disparate racial impacts and that choices in its development have led to a product that is unevenly useful to people of different races. It's also clear because facial recognition systems perform worse across all measures on people with darker skin (not just on this metric) - it's not like the accuracy rate is the same but the mistakes are different. Also, of course, yes - people will point out that different skin tones interact differently with photographic technology and algorithm designers are fighting an up hill battle. It's true! We have historically been good at solving the challenges of capturing images of light skinned people better than dark skinned ones. That's racist too! It's hard to not build a racist system when your tools already have racial bias. Again - pointing out the racial bias in this system is not an indictment of the people who built it. It's an indictment of the world it emerged out of and a bookmark to come back to when you see people using the outputs of the system.

lisasays 3y ago

It's idiotic to call it "racist", it's a shitty classifier This is sort of like telling residents of a historically black neighborhood which just so happens to have been exposed to drinking water with dangerously high levels of lead since time immemorial (despite decades of complaints) that it's just "shitty pipes", and that it's "idiotic" for anyone to think this state of affairs could possibly have something to do with racism. I agree that the matter is quite nuanced, and there was no racist intent. However it's far from "idiotic" to address these issues directly. If anything it seems a bit obtuse not to address them. Where there's smoke, after all, there's usually some form of fire.

prepend 3y ago
I think the lead pipes example is different because both the neighborhood structure and the lead are results of the same intentional racism. The classifier is not the result of any racist action on behalf of the programmer. It’s more like blaming a hospital administrator when people of a discriminated race have bad outcomes at their hospital, when the outcome is the systemic rate of all the other hospitals. The hospital outcome isn’t racist, because the systemic issues that affect the outcomes of the hospital patients have nothing to do with the decisions of the hospital.
- lisasays 3y ago
  Not necessarily -- recall that lead service lines were extremely widely used in the U.S. in all neighborhoods, black and white. So in fact it's more likely that he pipes being first put into the ground without any racist intent at all. It's the fact of this problem not being remediated in certain neighborhoods that cause people to ask, very legitimately -- how this came to pass.