3 ms·

I'm not sure I buy this interpretation. As the article says, these are "identity mentions", not definitive classifications of the author's identity. Don't these

I'm not sure I buy this interpretation. As the article says, these are "identity mentions", not definitive classifications of the author's identity. Don't these results just indicate that some identity labels are more likely to be used in offensive ways?

sdenton4 5y ago
‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that contain a token from a banned word list, disproportionately removes documents in dialects of English associated with minority identities (e.g., text in African American English, text discussing LGBTQ+ identities).’ A word so lovely that we really really don't want to risk putting it in our report...
- truffdog 5y ago
  If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.
- SpicyLemonZest 5y ago
  Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.
  - beerandt 5y ago
    It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.
- oh_sigh 5y ago
  To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.
- meragrin_ 5y ago
  > A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?
cratermoon 5y ago
Or... it could mean that the dominant culture defaults to describing identity labels as offensive. The labels themselves would only be offensive if someone deems them so. Remember when "gay" was a slur, but now it's an acceptable term for homosexual? That is, unless you're an edgy 13yo boy on the internet and think calling something 'gay' is funny.
- SpicyLemonZest 5y ago
  Again, this data doesn't show that the labels are offensive, just that they're more likely to be used in text which was classified as offensive. If there's lots of 13 year old boys on the Internet using "gay" as an insult, filtering out those insults from the corpus (and it's gottta be correct to filter them) could fully explain the high PMI for the word "gay".
  - cratermoon 5y ago
    > used in text which was classified as offensive Yes, and determining if a text is offensive is a very socially and culturally determined. Who determines the offensiveness of the context, and how? To shift gears a bit, how many rap song lyrics were filtered out from the corpus, for example? Does use of the n-word make an entire text de facto offensive? What about references to the common name of the moth Lymantria dispar dispar? What about the football team from Washington, D.C.? A lot of this just feels like a repeat of the Net Nanny internet filter days, when keywords were idiotically filtered out.
    - SpicyLemonZest 5y ago
      These are all important questions, but I don't think the article meaningfully engages with any of them beyond noting their existence. A lot of people are in a position where they have to judge whether a text is offensive; if your system autofills racial slurs, and you try to explain that it's not a problem because in some contexts the slurs have been reclaimed, you're gonna get in serious trouble.
- imbnwa 5y ago

sdenton4 5y ago

‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that contain a token from a banned word list, disproportionately removes documents in dialects of English associated with minority identities (e.g., text in African American English, text discussing LGBTQ+ identities).’ A word so lovely that we really really don't want to risk putting it in our report...

truffdog 5y ago
If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.
SpicyLemonZest 5y ago
Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.
- beerandt 5y ago
  It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.
oh_sigh 5y ago
To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.
meragrin_ 5y ago
> A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?

cratermoon 5y ago

Or... it could mean that the dominant culture defaults to describing identity labels as offensive. The labels themselves would only be offensive if someone deems them so. Remember when "gay" was a slur, but now it's an acceptable term for homosexual? That is, unless you're an edgy 13yo boy on the internet and think calling something 'gay' is funny.

SpicyLemonZest 5y ago
Again, this data doesn't show that the labels are offensive, just that they're more likely to be used in text which was classified as offensive. If there's lots of 13 year old boys on the Internet using "gay" as an insult, filtering out those insults from the corpus (and it's gottta be correct to filter them) could fully explain the high PMI for the word "gay".
- cratermoon 5y ago
  > used in text which was classified as offensive Yes, and determining if a text is offensive is a very socially and culturally determined. Who determines the offensiveness of the context, and how? To shift gears a bit, how many rap song lyrics were filtered out from the corpus, for example? Does use of the n-word make an entire text de facto offensive? What about references to the common name of the moth Lymantria dispar dispar? What about the football team from Washington, D.C.? A lot of this just feels like a repeat of the Net Nanny internet filter days, when keywords were idiotically filtered out.
  - SpicyLemonZest 5y ago
    These are all important questions, but I don't think the article meaningfully engages with any of them beyond noting their existence. A lot of people are in a position where they have to judge whether a text is offensive; if your system autofills racial slurs, and you try to explain that it's not a problem because in some contexts the slurs have been reclaimed, you're gonna get in serious trouble.
imbnwa 5y ago