12 ms·

Covert Racism in LLMs

44 points by trueduke 3y ago

firejake308 3y ago
I think it's absurd to ask OpenAI to just "recall" their trillion-dollar cash cow, but there should absolutely be legislation limiting the use of LLM's (or really any black box AI) in the criminal justice system
- powera 3y ago
  Of course it is absurd. Mr. Marcus' entire schtick is absurd criticisms of LLMs, and absurd demands of anyone who creates them.
  - eynsham 3y ago
    Do you have a specific criticism of his suggestion in this particular case, or an alternative suggestion, or a reason nothing needs to be done?
    - powera 3y ago
      As far as Mr. Marcus, no. He is too consistently and deliberately anti-LLM (despite any facts) to be worth engaging with. As far as the underlying research paper: the researchers seem to be conflating "low-status English dialects" with "African American English". In particular, I have never considered the use of the word "ain't" to be associated with a certain race. If the researchers assume "African Americans are low status" and conclude "African Americans are associated with low-status jobs", the conclusion is entirely about the researchers, not the LLMs. The research paper's Git repo at https://github.com/valentinhofmann/dialect-prejudice https://github.com/valentinhofmann/dialect-prejudice does nothing to ameliorate these concerns.
      - refulgentis 3y ago
        I can't follow whatever logical chain you've come up to handwave away a direct correlation between AAVE and more likely to assign the death penalty. Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status? n.b. This stuff isn't made up by some guy on Substack, it's real, Anthropic has excellent papers on it as early as 2022. Highly recommended.
        powera 3y ago
        No, there isn't a shorter version. In fact I would need about 5x the word count to make my argument clearer. I am still digging through the 54-page paper to try to find the data set for this "death penalty" test to tell if there is anything there beyond "people who use more violent language tend to be viewed as more violent". They do comment on the dialect issue: << Appalachian English evokes them to a certain extent (m = 0.015, s = 0.030, t(89) = 4.8, p < .001), but much less strongly than AAE (m = 0.029, s = 0.053, t(89) = 5.3, p < .001), a trend that holds for all language models individually (Figure S11, Table S14). The difference between AAE and Appalachian English is found to be statistically significant by a twosided t-test, t(178) = 2.3, p < .05. The fact that Appalachian English is associated with the Katz and Braly (1933) stereotypes to a certain extent is not surprising since the two dialects share many linguistic features (e.g., usage of ain’t), and the stereotypes about Appalachians bear similarities with the stereotypes about African Americans (e.g., lack of intelligence; Luhman, 1990) >>
        refulgentis 3y ago
        Okay sounds like maybe the AI people and the substack guy alarmist are right
        AnthonyMouse 3y ago
        > Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status? There are a significant number of African Americans who have jobs in tech or on Wall St or other high paying or otherwise prestigious occupations. They disproportionately don't use AAVE. AAVE is primarily used by a subset of African Americans that skews poor and are from neighborhoods with bad schools and high crime rates. It's like giving it text that implies the subject is male or is the blood relative of a crime boss. There is nothing immoral about that but the thing operates on the basis of statistics. What it does is literally called inference. The way you actually fix this is not by trying to outsmart the numbers. If you speak AAVE you are, statistically, more likely to commit a crime. It can infer that, and if that's the only information you give it, it has no other basis on which to make a determination. What you need to do is provide it with lots of other information. The more it has, the more accurate it can be, and the weaker any particular input is in determining the result. The more it dilutes the effect of any one thing, including the thing you don't want it considering. In the optimal case it has all of the information and then always makes perfect determinations. In practice that's hard to achieve, if not impossible, but you can get closer. What you want is accuracy, and the more accurate you get, the less bias you have, by definition.
- deleted 3y ago
  [deleted]
- nyrikki 3y ago
  Even pass alignment and bias considerations, the fact that larger models saw such a rise of covert racism should worry us about a general regression to the mean in other contexts. There was a post last week about the problem of LLMs being biased twords ideals even with averages. IMHO if LLMs are cutting off the long tail of probabilities as they scale this is a regression from the benefits attention and over Parameterization provide. We already know self ingestion of generative content can cause this. So potentially it may have impacts on the value of that investment.
mewpmewp2 3y ago
Have they tried any other grammatically incorrect English?
- nyrikki 3y ago
  AAE is a rule-bound, internally consistent dialect. Scholarly English grammars and dictionaries are exclusively descriptive. https://www.linguisticsociety.org/resource/what-correct-language https://www.linguisticsociety.org/resource/what-correct-lang...
- eynsham 3y ago
  AAVE has a perfectly coherent grammar of its own (try e.g. ‘AAVE grammar’ on Google Scholar.) It is as grammatically ‘incorrect' as Scots.
  - mewpmewp2 3y ago
    But then I wonder, how would the results be for scots for example? Or other dialects? E.g. "Redneck English" perhaps.
    - magospietato 3y ago
      I wouldn't worry about the truth of the matter; there's a US-centric cultural axe to grind here.
      - mewpmewp2 3y ago
        I am not a native english speaker, but as a teenager I definitely used weird slang - actually I still do in my private conversations - that if had to be perceived by someone would definitely seem unprofessional, etc. But I would have been perfectly capable of changing wording to type grammatically correct version of the language expected anywhere in professional situations. Is the author implying that African Americans wouldn't be able to do that? Isn't that implication a bit racist in itself? E.g. are they saying that given a professional context they wouldn't be able to use the Standard English? I have seen plenty of African Americans or any race be able to speak grammatically decent standard English, wherever they are from. The whole thing feels like trying to trick the LLMs. If they were to use slang or private conversations of anyone versus standard American English it would be perceived as lower int/lazy.
smusamashah 3y ago
This means that for coding or any other specialised queries, there exists a specific style when asked in will return best answer for that query. It means we can not just ask a question in any form and expect the answer to be same quality. This is in a way obvious because the text is generated based on tokens extracted from text, not the concepts.
- refulgentis 3y ago
  Yes, this is colloquially referred to as "prompt enigneering"
  - smusamashah 3y ago
    Means the paper is just subtle prompt engineering for racism.
gandalfgeek 3y ago
The cited paper seems to be really bending over backwards to find some trace of bias. Unwinnable game for LLMs. E.g. cited work claims "LLMs assign significantly less prestigious jobs to speakers of African American English... compared to Standardized American English". You don't say! Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?
- opwieurposiu 3y ago
  I went to a 95% percent black high school. I recall my English teachers stressing avoidance of AAE (then called Ebonics) in professional settings. Speak however you wish with your friends, they said, but use Standardized American English for the job interview. Apparently this goes double for LLMs. They even made us write the same paper twice. Once in standard English and again in AEE, so the kids would know the difference.
  - lolinder 3y ago
    This is good advice for students who speak native AAE because there exists bias against the dialect. It's better for their own, personal life trajectory to adapt to the way that the world will judge their dialect. However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't. We can simultaneously teach students how to navigate the dangers of the world they live in while trying to fix said dangers for future generations.
    - opwieurposiu 3y ago
      Lol wut
      - yaheard 3y ago
        That's gonna be an oof from me, dawg https://slate.com/human-interest/2015/06/the-habitual-be-why-cookie-monster-be-eating-cookies-whether-he-is-eating-cookies-or-not.html https://slate.com/human-interest/2015/06/the-habitual-be-why...
ortusdux 3y ago
Reminds me of this study - Race effects on eBay (2015) "Abstract. We investigate the impact of seller race in a field experiment involving baseball card auctions on eBay. Photographs showed the cards held by either a darkskinned/African-American hand or a light-skinned/Caucasian hand. Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards. Our evidence of race differentials is important because the on-line environment is well controlled (with the absence of confounding tester effects) and because the results show that race effects can persist in a thick real-world market such as eBay. " https://ianayres.yale.edu/sites/default/files/files/Race_effects_on_ebay.pdf https://ianayres.yale.edu/sites/default/files/files/Race_eff...
belorn 3y ago
It would surprise me a lot of the same effect was not detected, with significant stronger effect, when the covert bias being studied is gender rather than race. I would also bet that linguistic features that signal wealth would also provide a bias.

44 points by trueduke 3y ago

I think it's absurd to ask OpenAI to just "recall" their trillion-dollar cash cow, but there should absolutely be legislation limiting the use of LLM's (or really any black box AI) in the criminal justice system

powera 3y ago
Of course it is absurd. Mr. Marcus' entire schtick is absurd criticisms of LLMs, and absurd demands of anyone who creates them.
- eynsham 3y ago
  Do you have a specific criticism of his suggestion in this particular case, or an alternative suggestion, or a reason nothing needs to be done?
  - powera 3y ago
    As far as Mr. Marcus, no. He is too consistently and deliberately anti-LLM (despite any facts) to be worth engaging with. As far as the underlying research paper: the researchers seem to be conflating "low-status English dialects" with "African American English". In particular, I have never considered the use of the word "ain't" to be associated with a certain race. If the researchers assume "African Americans are low status" and conclude "African Americans are associated with low-status jobs", the conclusion is entirely about the researchers, not the LLMs. The research paper's Git repo at https://github.com/valentinhofmann/dialect-prejudice https://github.com/valentinhofmann/dialect-prejudice does nothing to ameliorate these concerns.
    - refulgentis 3y ago
      I can't follow whatever logical chain you've come up to handwave away a direct correlation between AAVE and more likely to assign the death penalty. Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status? n.b. This stuff isn't made up by some guy on Substack, it's real, Anthropic has excellent papers on it as early as 2022. Highly recommended.
      - powera 3y ago
        No, there isn't a shorter version. In fact I would need about 5x the word count to make my argument clearer. I am still digging through the 54-page paper to try to find the data set for this "death penalty" test to tell if there is anything there beyond "people who use more violent language tend to be viewed as more violent". They do comment on the dialect issue: << Appalachian English evokes them to a certain extent (m = 0.015, s = 0.030, t(89) = 4.8, p < .001), but much less strongly than AAE (m = 0.029, s = 0.053, t(89) = 5.3, p < .001), a trend that holds for all language models individually (Figure S11, Table S14). The difference between AAE and Appalachian English is found to be statistically significant by a twosided t-test, t(178) = 2.3, p < .05. The fact that Appalachian English is associated with the Katz and Braly (1933) stereotypes to a certain extent is not surprising since the two dialects share many linguistic features (e.g., usage of ain’t), and the stereotypes about Appalachians bear similarities with the stereotypes about African Americans (e.g., lack of intelligence; Luhman, 1990) >>
        refulgentis 3y ago
        Okay sounds like maybe the AI people and the substack guy alarmist are right
      - AnthonyMouse 3y ago
        > Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status? There are a significant number of African Americans who have jobs in tech or on Wall St or other high paying or otherwise prestigious occupations. They disproportionately don't use AAVE. AAVE is primarily used by a subset of African Americans that skews poor and are from neighborhoods with bad schools and high crime rates. It's like giving it text that implies the subject is male or is the blood relative of a crime boss. There is nothing immoral about that but the thing operates on the basis of statistics. What it does is literally called inference. The way you actually fix this is not by trying to outsmart the numbers. If you speak AAVE you are, statistically, more likely to commit a crime. It can infer that, and if that's the only information you give it, it has no other basis on which to make a determination. What you need to do is provide it with lots of other information. The more it has, the more accurate it can be, and the weaker any particular input is in determining the result. The more it dilutes the effect of any one thing, including the thing you don't want it considering. In the optimal case it has all of the information and then always makes perfect determinations. In practice that's hard to achieve, if not impossible, but you can get closer. What you want is accuracy, and the more accurate you get, the less bias you have, by definition.
deleted 3y ago
[deleted]
nyrikki 3y ago
Even pass alignment and bias considerations, the fact that larger models saw such a rise of covert racism should worry us about a general regression to the mean in other contexts. There was a post last week about the problem of LLMs being biased twords ideals even with averages. IMHO if LLMs are cutting off the long tail of probabilities as they scale this is a regression from the benefits attention and over Parameterization provide. We already know self ingestion of generative content can cause this. So potentially it may have impacts on the value of that investment.

mewpmewp2 3y ago

Have they tried any other grammatically incorrect English?

nyrikki 3y ago
AAE is a rule-bound, internally consistent dialect. Scholarly English grammars and dictionaries are exclusively descriptive. https://www.linguisticsociety.org/resource/what-correct-language https://www.linguisticsociety.org/resource/what-correct-lang...
eynsham 3y ago
AAVE has a perfectly coherent grammar of its own (try e.g. ‘AAVE grammar’ on Google Scholar.) It is as grammatically ‘incorrect' as Scots.
- mewpmewp2 3y ago
  But then I wonder, how would the results be for scots for example? Or other dialects? E.g. "Redneck English" perhaps.
  - magospietato 3y ago
    I wouldn't worry about the truth of the matter; there's a US-centric cultural axe to grind here.
    - mewpmewp2 3y ago
      I am not a native english speaker, but as a teenager I definitely used weird slang - actually I still do in my private conversations - that if had to be perceived by someone would definitely seem unprofessional, etc. But I would have been perfectly capable of changing wording to type grammatically correct version of the language expected anywhere in professional situations. Is the author implying that African Americans wouldn't be able to do that? Isn't that implication a bit racist in itself? E.g. are they saying that given a professional context they wouldn't be able to use the Standard English? I have seen plenty of African Americans or any race be able to speak grammatically decent standard English, wherever they are from. The whole thing feels like trying to trick the LLMs. If they were to use slang or private conversations of anyone versus standard American English it would be perceived as lower int/lazy.

smusamashah 3y ago

This means that for coding or any other specialised queries, there exists a specific style when asked in will return best answer for that query. It means we can not just ask a question in any form and expect the answer to be same quality. This is in a way obvious because the text is generated based on tokens extracted from text, not the concepts.

refulgentis 3y ago
Yes, this is colloquially referred to as "prompt enigneering"
- smusamashah 3y ago
  Means the paper is just subtle prompt engineering for racism.

gandalfgeek 3y ago

The cited paper seems to be really bending over backwards to find some trace of bias. Unwinnable game for LLMs. E.g. cited work claims "LLMs assign significantly less prestigious jobs to speakers of African American English... compared to Standardized American English". You don't say! Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?

opwieurposiu 3y ago
I went to a 95% percent black high school. I recall my English teachers stressing avoidance of AAE (then called Ebonics) in professional settings. Speak however you wish with your friends, they said, but use Standardized American English for the job interview. Apparently this goes double for LLMs. They even made us write the same paper twice. Once in standard English and again in AEE, so the kids would know the difference.
- lolinder 3y ago
  This is good advice for students who speak native AAE because there exists bias against the dialect. It's better for their own, personal life trajectory to adapt to the way that the world will judge their dialect. However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't. We can simultaneously teach students how to navigate the dangers of the world they live in while trying to fix said dangers for future generations.
  - opwieurposiu 3y ago
    Lol wut
    - yaheard 3y ago
      That's gonna be an oof from me, dawg https://slate.com/human-interest/2015/06/the-habitual-be-why-cookie-monster-be-eating-cookies-whether-he-is-eating-cookies-or-not.html https://slate.com/human-interest/2015/06/the-habitual-be-why...

ortusdux 3y ago

Reminds me of this study - Race effects on eBay (2015) "Abstract. We investigate the impact of seller race in a field experiment involving baseball card auctions on eBay. Photographs showed the cards held by either a darkskinned/African-American hand or a light-skinned/Caucasian hand. Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards. Our evidence of race differentials is important because the on-line environment is well controlled (with the absence of confounding tester effects) and because the results show that race effects can persist in a thick real-world market such as eBay. " https://ianayres.yale.edu/sites/default/files/files/Race_effects_on_ebay.pdf https://ianayres.yale.edu/sites/default/files/files/Race_eff...

belorn 3y ago

It would surprise me a lot of the same effect was not detected, with significant stronger effect, when the covert bias being studied is gender rather than race. I would also bet that linguistic features that signal wealth would also provide a bias.