5 ms·

Minority voices ‘filtered’ out of Google Natural Language Processing models

47 points by Hard_Space 5y ago

solarmist 5y ago
‘Our examination of the excluded data suggests that documents associated with Black and Hispanic authors and documents mentioning sexual orientations are signiﬁcantly more likely to be excluded by C4.EN’s blocklist filtering, and that many excluded documents contained non-offensive or non-sexual content (e.g., legislative discussions of same-sex marriage, scientiﬁc and medical content).’ Very disheartening, but I'm not surprised.
- fIoat 5y ago
  "I don't have a problem with those people, but my grandma/the Chinese market/our stakeholders might, so it is a risk I cannot take." Same old story, history repeats. We are just cementing the abstract into code.
SpicyLemonZest 5y ago
I'm not sure I buy this interpretation. As the article says, these are "identity mentions", not definitive classifications of the author's identity. Don't these results just indicate that some identity labels are more likely to be used in offensive ways?
- sdenton4 5y ago
  ‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that contain a token from a banned word list, disproportionately removes documents in dialects of English associated with minority identities (e.g., text in African American English, text discussing LGBTQ+ identities).’ A word so lovely that we really really don't want to risk putting it in our report...
  - truffdog 5y ago
    If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.
  - SpicyLemonZest 5y ago
    Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.
    - beerandt 5y ago
      It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.
  - oh_sigh 5y ago
    To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.
  - meragrin_ 5y ago
    > A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?
- cratermoon 5y ago
  Or... it could mean that the dominant culture defaults to describing identity labels as offensive. The labels themselves would only be offensive if someone deems them so. Remember when "gay" was a slur, but now it's an acceptable term for homosexual? That is, unless you're an edgy 13yo boy on the internet and think calling something 'gay' is funny.
  - SpicyLemonZest 5y ago
    Again, this data doesn't show that the labels are offensive, just that they're more likely to be used in text which was classified as offensive. If there's lots of 13 year old boys on the Internet using "gay" as an insult, filtering out those insults from the corpus (and it's gottta be correct to filter them) could fully explain the high PMI for the word "gay".
    - cratermoon 5y ago
      > used in text which was classified as offensive Yes, and determining if a text is offensive is a very socially and culturally determined. Who determines the offensiveness of the context, and how? To shift gears a bit, how many rap song lyrics were filtered out from the corpus, for example? Does use of the n-word make an entire text de facto offensive? What about references to the common name of the moth Lymantria dispar dispar? What about the football team from Washington, D.C.? A lot of this just feels like a repeat of the Net Nanny internet filter days, when keywords were idiotically filtered out.
      - SpicyLemonZest 5y ago
        These are all important questions, but I don't think the article meaningfully engages with any of them beyond noting their existence. A lot of people are in a position where they have to judge whether a text is offensive; if your system autofills racial slurs, and you try to explain that it's not a problem because in some contexts the slurs have been reclaimed, you're gonna get in serious trouble.
  - imbnwa 5y ago
AlbertCory 5y ago
Just a note on something I do have personal experience with: The authors also observed that the text of many patents are initially obtained via imperfect examples of Optical Character Recognition (OCR), with their accompanying errors in English possibly passed through to the C4 data with little or no annotation that would distinguish it from acceptable English. While patent offices do release PDFs of all their patent docs (and it isn't just patents; it's all the back-and-forth between the examiners & the applicant, too), a huge percentage are images of paper documents. You can always tell just by double-clicking on a word -- if the word doesn't highlight, it's an image. OCR output from these things is generally terrible. There is no way it should ever be input to any ML model.
have_faith 5y ago
The first paragraph is written to imply they have been filtered purposely: > has been extensively ‘filtered’ to remove black and Hispanic authors, as well as material related to gay and lesbian identities But after it is explained that this is an unintended second order effect from trying to remove offensive content from the corpus. I'm not trying to justify the outcome (it's an issue regardless of intent) I just didn't think there was any need to strongly imply it was intended.
- cratermoon 5y ago
  > an unintended second order effect from trying to remove offensive content from the corpus. How odd that "offensive" and "minority" turned out to overlap so much. How could that have possibly happened?
  - User23 5y ago
    Use of the n-word is an obvious example.
    - tablespoon 5y ago
      Also "minority" tends to be associated with a lower socioeconomic status, which may imply cruder language with more swear words in general, etc.
    - imbnwa 5y ago
      Which Black folk use colliquially, either to imply that the addressee needs to be more humilitous or apply greater self-improvement about a subject at hand, or to re-enforce camraderie.
- brudgers 5y ago
  It was intentional at least in so far as that bias is a known problem with Google's machine learning processes, and that no substantial effort was made to avoid the bias. Running over a pedestrian might be a second order effect of driving a car, except that driving a car includes applying the brakes to avoid it.
d1a2n 5y ago
The greatest human rights issue of our time is LGBTQ voices being heard. I know it might sound like hyperbole but the threat the community is under from a resurgence of antiquated notions of sexual morality, accusations of a "slippery slope into pedophilia acceptance", and other such nonsense is in my estimation an existential threat. The disgusting notion that this has anything to do with perversion or mental illness is something that we need to take very seriously. The trans community in particular I believe is especially vulnerable. We see incredible numbers, 4000%, 10000% increases in young people looking for gender reassignment therapy, and it's beautiful, but also terrifying to consider what will probably be a concomitant increase in intolerance and bigotry, as those who only know hate seem to have a limitless reserve. Good on Google for focusing almost all of their AI ethics work on debiasing, because we sure as hell need it. They tell us they no longer stand by "don't be evil", but I'm not so sure of that.
- thefounder 5y ago
  >> The greatest human rights issue of our time is LGBTQ voices being heard. Maybe in the western world but we know that there are places where women have really no voice (or less) or places like NK where very few people have a voice...there is also the forced labor in China and Africa. I highly doubt the greatest human rights issue of our time is LBQT voices not being heard. That could only be said by someone who thinks less for other people in need. Minorities are always "heard" less but let's not make this the greatest human rights issue. People are being beheaded and people are dying of starvation right now. They've been filtered out by the whole world.
  - deleted 5y ago
    [deleted]
  - d1a2n 5y ago
    If I could I would hook you up with your eyes held open Clockwork Orange style and make you watch Blues Clues gender parade on repeat until you understand the pain of trans oppression.
    - 5y ago
cratermoon 5y ago
I'm shocked, SHOCKED to find racism and white supremacy reflected in Silicon Valley software.
endymi0n 5y ago
Sounds to me like a "damned if you do, damned if you don't" basically impossible task without making ML aware of cultural intricacies, trigger warnings, age appropriate approaches and parental controls at the same time. Like, it's not like this filtering was done for the purpose of silencing anyone — Google (among others) really learned the hard way to not feed smut to ML models, as it _will_ get regurgitated, always as a possible PR disaster in the making: https://www.buzzfeed.com/fionarutherford/heres-why-some-people-think-googles-results-are-racist https://www.buzzfeed.com/fionarutherford/heres-why-some-peop... https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n_56f3e678e4b04c4c37615502 https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n... Still interesting research, but playing the devil's advocate, I think I can see why this part of the corpus was more extensively filtered away. Sad issue, but I don't think we'd have a sane way out without massive, manual whitelisting.
- cyanydeez 5y ago
  actually, if your goal is "handle 99.9% of circumstances", your job is to find the edge cases and make sure they are as normal as all other cases. thats not whats happening in ML. whats happening is "find the easiest amount of information to train a model"
jsnell 5y ago
I believe this is the word list that authors are objecting to the use of: https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/en https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and... The list does seem a bit... umm... oddly specific in places, probably due to its history as being first compiled for a photo sharing site.
- s1artibartfast 5y ago
  This is great! It has been a long time since I learned new dirty worlds. The inclusion of misspelled words and popular culture references is also intriguing.
- inglor_cz 5y ago
  Until today, the only thing I could imagine under "Alaskan pipeline" was, well, Alaskan pipeline. How naive I was.
crumpled 5y ago
Why would someone want to curate their data if they want to build a good model? Are they worried that a scientist is going to be exposed to a bad word? Are they imposing some cultural brand of morality upon the AI?

47 points by Hard_Space 5y ago

‘Our examination of the excluded data suggests that documents associated with Black and Hispanic authors and documents mentioning sexual orientations are signiﬁcantly more likely to be excluded by C4.EN’s blocklist filtering, and that many excluded documents contained non-offensive or non-sexual content (e.g., legislative discussions of same-sex marriage, scientiﬁc and medical content).’ Very disheartening, but I'm not surprised.

fIoat 5y ago
"I don't have a problem with those people, but my grandma/the Chinese market/our stakeholders might, so it is a risk I cannot take." Same old story, history repeats. We are just cementing the abstract into code.

SpicyLemonZest 5y ago

I'm not sure I buy this interpretation. As the article says, these are "identity mentions", not definitive classifications of the author's identity. Don't these results just indicate that some identity labels are more likely to be used in offensive ways?

sdenton4 5y ago
‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that contain a token from a banned word list, disproportionately removes documents in dialects of English associated with minority identities (e.g., text in African American English, text discussing LGBTQ+ identities).’ A word so lovely that we really really don't want to risk putting it in our report...
- truffdog 5y ago
  If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.
- SpicyLemonZest 5y ago
  Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.
  - beerandt 5y ago
    It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.
- oh_sigh 5y ago
  To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.
- meragrin_ 5y ago
  > A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?
cratermoon 5y ago
Or... it could mean that the dominant culture defaults to describing identity labels as offensive. The labels themselves would only be offensive if someone deems them so. Remember when "gay" was a slur, but now it's an acceptable term for homosexual? That is, unless you're an edgy 13yo boy on the internet and think calling something 'gay' is funny.
- SpicyLemonZest 5y ago
  Again, this data doesn't show that the labels are offensive, just that they're more likely to be used in text which was classified as offensive. If there's lots of 13 year old boys on the Internet using "gay" as an insult, filtering out those insults from the corpus (and it's gottta be correct to filter them) could fully explain the high PMI for the word "gay".
  - cratermoon 5y ago
    > used in text which was classified as offensive Yes, and determining if a text is offensive is a very socially and culturally determined. Who determines the offensiveness of the context, and how? To shift gears a bit, how many rap song lyrics were filtered out from the corpus, for example? Does use of the n-word make an entire text de facto offensive? What about references to the common name of the moth Lymantria dispar dispar? What about the football team from Washington, D.C.? A lot of this just feels like a repeat of the Net Nanny internet filter days, when keywords were idiotically filtered out.
    - SpicyLemonZest 5y ago
      These are all important questions, but I don't think the article meaningfully engages with any of them beyond noting their existence. A lot of people are in a position where they have to judge whether a text is offensive; if your system autofills racial slurs, and you try to explain that it's not a problem because in some contexts the slurs have been reclaimed, you're gonna get in serious trouble.
- imbnwa 5y ago

AlbertCory 5y ago

Just a note on something I do have personal experience with: The authors also observed that the text of many patents are initially obtained via imperfect examples of Optical Character Recognition (OCR), with their accompanying errors in English possibly passed through to the C4 data with little or no annotation that would distinguish it from acceptable English. While patent offices do release PDFs of all their patent docs (and it isn't just patents; it's all the back-and-forth between the examiners & the applicant, too), a huge percentage are images of paper documents. You can always tell just by double-clicking on a word -- if the word doesn't highlight, it's an image. OCR output from these things is generally terrible. There is no way it should ever be input to any ML model.

have_faith 5y ago

The first paragraph is written to imply they have been filtered purposely: > has been extensively ‘filtered’ to remove black and Hispanic authors, as well as material related to gay and lesbian identities But after it is explained that this is an unintended second order effect from trying to remove offensive content from the corpus. I'm not trying to justify the outcome (it's an issue regardless of intent) I just didn't think there was any need to strongly imply it was intended.

cratermoon 5y ago
> an unintended second order effect from trying to remove offensive content from the corpus. How odd that "offensive" and "minority" turned out to overlap so much. How could that have possibly happened?
- User23 5y ago
  Use of the n-word is an obvious example.
  - tablespoon 5y ago
    Also "minority" tends to be associated with a lower socioeconomic status, which may imply cruder language with more swear words in general, etc.
  - imbnwa 5y ago
    Which Black folk use colliquially, either to imply that the addressee needs to be more humilitous or apply greater self-improvement about a subject at hand, or to re-enforce camraderie.
brudgers 5y ago
It was intentional at least in so far as that bias is a known problem with Google's machine learning processes, and that no substantial effort was made to avoid the bias. Running over a pedestrian might be a second order effect of driving a car, except that driving a car includes applying the brakes to avoid it.

d1a2n 5y ago

The greatest human rights issue of our time is LGBTQ voices being heard. I know it might sound like hyperbole but the threat the community is under from a resurgence of antiquated notions of sexual morality, accusations of a "slippery slope into pedophilia acceptance", and other such nonsense is in my estimation an existential threat. The disgusting notion that this has anything to do with perversion or mental illness is something that we need to take very seriously. The trans community in particular I believe is especially vulnerable. We see incredible numbers, 4000%, 10000% increases in young people looking for gender reassignment therapy, and it's beautiful, but also terrifying to consider what will probably be a concomitant increase in intolerance and bigotry, as those who only know hate seem to have a limitless reserve. Good on Google for focusing almost all of their AI ethics work on debiasing, because we sure as hell need it. They tell us they no longer stand by "don't be evil", but I'm not so sure of that.

thefounder 5y ago
>> The greatest human rights issue of our time is LGBTQ voices being heard. Maybe in the western world but we know that there are places where women have really no voice (or less) or places like NK where very few people have a voice...there is also the forced labor in China and Africa. I highly doubt the greatest human rights issue of our time is LBQT voices not being heard. That could only be said by someone who thinks less for other people in need. Minorities are always "heard" less but let's not make this the greatest human rights issue. People are being beheaded and people are dying of starvation right now. They've been filtered out by the whole world.
- deleted 5y ago
  [deleted]
- d1a2n 5y ago
  If I could I would hook you up with your eyes held open Clockwork Orange style and make you watch Blues Clues gender parade on repeat until you understand the pain of trans oppression.
  - 5y ago

cratermoon 5y ago

I'm shocked, SHOCKED to find racism and white supremacy reflected in Silicon Valley software.

endymi0n 5y ago

Sounds to me like a "damned if you do, damned if you don't" basically impossible task without making ML aware of cultural intricacies, trigger warnings, age appropriate approaches and parental controls at the same time. Like, it's not like this filtering was done for the purpose of silencing anyone — Google (among others) really learned the hard way to not feed smut to ML models, as it _will_ get regurgitated, always as a possible PR disaster in the making: https://www.buzzfeed.com/fionarutherford/heres-why-some-people-think-googles-results-are-racist https://www.buzzfeed.com/fionarutherford/heres-why-some-peop... https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n_56f3e678e4b04c4c37615502 https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n... Still interesting research, but playing the devil's advocate, I think I can see why this part of the corpus was more extensively filtered away. Sad issue, but I don't think we'd have a sane way out without massive, manual whitelisting.

cyanydeez 5y ago
actually, if your goal is "handle 99.9% of circumstances", your job is to find the edge cases and make sure they are as normal as all other cases. thats not whats happening in ML. whats happening is "find the easiest amount of information to train a model"

jsnell 5y ago

I believe this is the word list that authors are objecting to the use of: https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/en https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and... The list does seem a bit... umm... oddly specific in places, probably due to its history as being first compiled for a photo sharing site.

s1artibartfast 5y ago
This is great! It has been a long time since I learned new dirty worlds. The inclusion of misspelled words and popular culture references is also intriguing.
inglor_cz 5y ago
Until today, the only thing I could imagine under "Alaskan pipeline" was, well, Alaskan pipeline. How naive I was.

crumpled 5y ago

Why would someone want to curate their data if they want to build a good model? Are they worried that a scientist is going to be exposed to a bad word? Are they imposing some cultural brand of morality upon the AI?