4 ms·

‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that cont

‘Some ﬁlters are relatively straightforward, such as removing Lorem ipsum placeholder text. However, we ﬁnd that another ﬁlter which removes documents that contain a token from a banned word list, disproportionately removes documents in dialects of English associated with minority identities (e.g., text in African American English, text discussing LGBTQ+ identities).’ A word so lovely that we really really don't want to risk putting it in our report...

truffdog 5y ago
If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.
SpicyLemonZest 5y ago
Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.
- beerandt 5y ago
  It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.
oh_sigh 5y ago
To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.
meragrin_ 5y ago
> A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?

truffdog 5y ago

If they didn't filter those words out, then presumably the headline would be about 'perpetuating' instead of about 'filtering'.

SpicyLemonZest 5y ago

Yeah, I have similar concerns about the different dialects thing. It's true that "the meaning of seemingly 'bad' words heavily depends on the social context", but most realistic systems are gonna be deployed in multiple social contexts and ideally shouldn't be offensive in any of them.

beerandt 5y ago
It's not that there are different contexts, it's that with particular identity groups, the "rules" for that context vary so much over time (and context) as well. For most of the 90's saying black was "wrong" in favor of African American. Now we've gone full circle, and the same people who made a sour-face at black 25 years ago are capitalizing it. That's probably the most benign, easy case. Of course ML can't keep up with loaded-terms and slurs; most people can hardly keep track of it all.

oh_sigh 5y ago

To mirror society's usage of the word, "nigger"(and variants) can only be used in a dataset when the ML researchers developing the model are Black. Perhaps tech companies with mixed race ML groups can have a clean room set up which only Black people can access in order to do their work with this dangerous and powerful word.

meragrin_ 5y ago

> A word so lovely that we really really don't want to risk putting it in our report... I can't tell if that is sarcasm or not. Is the 'lovely word' disclosed somewhere else?