4 ms·

May be a weird takeaway, but I did find it strange how much the whole report focussed on misgendering as a safety issue. I agree it’s important to get right, b

by Closi 3y ago

May be a weird takeaway, but I did find it strange how much the whole report focussed on misgendering as a safety issue. I agree it’s important to get right, but it seems like one of hundreds of safety/alignment issues and that many others are de-emphasised or ignored.

hoseja 3y ago
[flagged]
vineyardmike 3y ago
I didn't find it to be a particularly notable issue relative to the rest of the issues they mentioned. It didn't seem to be overrepresented to me... That said, it's something that is more controllable across languages. All people, in all languages, have a roughly equal distribution of genders, but not race/religion, etc. Japanese language text will have similar gender distributions to English, but likely not equal distributions discussing race. That makes it a much better litmus test for multi-lingual bias. Most of the misgendering discussion (2-3 paragraphs?) was in the translation section, which makes sense. A lot of the first classes in foundation courses learning a foreign language revolved around pronouns (which don't work the same in every language). Gender may be implied or absent in some. For example, to say "she is a doctor" in Italian, you might say "è un dottore", which has no pronoun (literally "is a doctor"). If you use google translate to make it English, "he" is added, assuming the gender. The potential for bias here is obvious, but consider that LLMs often deal with more context than a single sentence - if you're translating or writing story about a female doctor (where the gender is available contextually), you want all the use of pronouns to align where it makes sense. If a LLM didn't "understand" the pronoun in Italian, you might not recognize it, but in English, if the same person's gender was mixed across sentences, it'd be hard to read.
- cowl 3y ago
  I'm not sure you got the right example here. "è un dottore" in Italian is unambiguously a He. Italian Language is a gendered one and leaves little room for context interpretation. A female doctor would be 'una dottoressa" and you would never say it without a pronoun either so the full frase "lei è una dottoressa" does leave zero room for implying anything. Maybe your comment would be more valid in other languages that are not so strongly gendered.
noname120 3y ago
Generalized artificial intelligence (AGI) comes with very real x-risks[1] (existential risks) and s-risks[2] (suffering risks). An expert survey of 738 researchers who published in NeurIPS and ICML was done last year[3]. Their median estimation that AI will have an “extremely bad” long term outcome is 5%, and 48% of the researchers estimate the probability to be at least 10%. This is worryingly high considering the absolutely catastrophic consequences of the scenarios. A minority of very vocal AI researchers (e.g. Yann LeCun) dismiss these risks entirely and claim that people read too much science fiction. But when you listen to their interviews it's very clear that they have no idea what they are talking about and never actually read any scientific literature on the subject. The study of AI risks is a serious area of academic research that is worked on by labs from Stanford[4], Berkeley[5], Carnegie Melon University[6], Oxford[7], Cambridge[8], and many MANY other universities[9]. Not people who read too much science fiction. ———— [1] https://arxiv.org/pdf/2206.05862.pdf https://arxiv.org/pdf/2206.05862.pdf [2] https://longtermrisk.org/files/Sotala-Gloor-Superintelligent-AI-and-Suffering-Risks.pdf#page=8&search=%22superintelligence%22 https://longtermrisk.org/files/Sotala-Gloor-Superintelligent... [3] https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ [4] https://web.stanford.edu/~chadj/existentialrisk.pdf https://web.stanford.edu/~chadj/existentialrisk.pdf [5] https://humancompatible.ai/about/ https://humancompatible.ai/about/ [6] https://www.cs.cmu.edu/~focal/ https://www.cs.cmu.edu/~focal/ [7] https://www.fhi.ox.ac.uk/research/research-areas/#aisafety_tab https://www.fhi.ox.ac.uk/research/research-areas/#aisafety_t... [8] https://www.camxrisk.org/ https://www.camxrisk.org/ [9] https://futureoflife.org/about-us/our-people/ai-existential-safety-community/ https://futureoflife.org/about-us/our-people/ai-existential-...
zarzavat 3y ago
Misgendering in translation is interesting not just because of wokeness but because it is an assuredly AI-complete subproblem in sometimes not AI-complete domain. For example, let’s say I want to translate “the cat sat on the mat” from English to French. This doesn’t require LLMs, the old bayesian Google Translate could do that just fine. Now let’s say you want to translate “Carol went to the store. They[3pp] bought some eggs” from a language that doesn’t have gendered 3rd person pronouns, to English which does have gendered pronouns. Now the model needs to know that Carol is a “she”, otherwise you will get the erroneous output “Carol went to the store. He bought some eggs.” Let’s say we have: “Obama went to the store. [3pp] bought some eggs”. Now the model needs to know whether we are referring to Barack Obama or Michelle Obama so it needs to look back in the context to figure out which Obama which requires comprehension and world knowledge. For example if we precede with “After attending the national security briefing, …” then the model needs to know that: 1) national security briefings are attended by Presidents, 2) Barack Obama was President, in order to deduce that 3) “Obama” here is a “He”. Getting pronouns right matching human performances requires that model understands language and has some knowledge of the world.

by Closi 3y ago

hoseja 3y ago

[flagged]

vineyardmike 3y ago

I didn't find it to be a particularly notable issue relative to the rest of the issues they mentioned. It didn't seem to be overrepresented to me... That said, it's something that is more controllable across languages. All people, in all languages, have a roughly equal distribution of genders, but not race/religion, etc. Japanese language text will have similar gender distributions to English, but likely not equal distributions discussing race. That makes it a much better litmus test for multi-lingual bias. Most of the misgendering discussion (2-3 paragraphs?) was in the translation section, which makes sense. A lot of the first classes in foundation courses learning a foreign language revolved around pronouns (which don't work the same in every language). Gender may be implied or absent in some. For example, to say "she is a doctor" in Italian, you might say "è un dottore", which has no pronoun (literally "is a doctor"). If you use google translate to make it English, "he" is added, assuming the gender. The potential for bias here is obvious, but consider that LLMs often deal with more context than a single sentence - if you're translating or writing story about a female doctor (where the gender is available contextually), you want all the use of pronouns to align where it makes sense. If a LLM didn't "understand" the pronoun in Italian, you might not recognize it, but in English, if the same person's gender was mixed across sentences, it'd be hard to read.

cowl 3y ago
I'm not sure you got the right example here. "è un dottore" in Italian is unambiguously a He. Italian Language is a gendered one and leaves little room for context interpretation. A female doctor would be 'una dottoressa" and you would never say it without a pronoun either so the full frase "lei è una dottoressa" does leave zero room for implying anything. Maybe your comment would be more valid in other languages that are not so strongly gendered.

noname120 3y ago

Generalized artificial intelligence (AGI) comes with very real x-risks[1] (existential risks) and s-risks[2] (suffering risks). An expert survey of 738 researchers who published in NeurIPS and ICML was done last year[3]. Their median estimation that AI will have an “extremely bad” long term outcome is 5%, and 48% of the researchers estimate the probability to be at least 10%. This is worryingly high considering the absolutely catastrophic consequences of the scenarios. A minority of very vocal AI researchers (e.g. Yann LeCun) dismiss these risks entirely and claim that people read too much science fiction. But when you listen to their interviews it's very clear that they have no idea what they are talking about and never actually read any scientific literature on the subject. The study of AI risks is a serious area of academic research that is worked on by labs from Stanford[4], Berkeley[5], Carnegie Melon University[6], Oxford[7], Cambridge[8], and many MANY other universities[9]. Not people who read too much science fiction. ———— [1] https://arxiv.org/pdf/2206.05862.pdf https://arxiv.org/pdf/2206.05862.pdf [2] https://longtermrisk.org/files/Sotala-Gloor-Superintelligent-AI-and-Suffering-Risks.pdf#page=8&search=%22superintelligence%22 https://longtermrisk.org/files/Sotala-Gloor-Superintelligent... [3] https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ [4] https://web.stanford.edu/~chadj/existentialrisk.pdf https://web.stanford.edu/~chadj/existentialrisk.pdf [5] https://humancompatible.ai/about/ https://humancompatible.ai/about/ [6] https://www.cs.cmu.edu/~focal/ https://www.cs.cmu.edu/~focal/ [7] https://www.fhi.ox.ac.uk/research/research-areas/#aisafety_tab https://www.fhi.ox.ac.uk/research/research-areas/#aisafety_t... [8] https://www.camxrisk.org/ https://www.camxrisk.org/ [9] https://futureoflife.org/about-us/our-people/ai-existential-safety-community/ https://futureoflife.org/about-us/our-people/ai-existential-...

zarzavat 3y ago

Misgendering in translation is interesting not just because of wokeness but because it is an assuredly AI-complete subproblem in sometimes not AI-complete domain. For example, let’s say I want to translate “the cat sat on the mat” from English to French. This doesn’t require LLMs, the old bayesian Google Translate could do that just fine. Now let’s say you want to translate “Carol went to the store. They[3pp] bought some eggs” from a language that doesn’t have gendered 3rd person pronouns, to English which does have gendered pronouns. Now the model needs to know that Carol is a “she”, otherwise you will get the erroneous output “Carol went to the store. He bought some eggs.” Let’s say we have: “Obama went to the store. [3pp] bought some eggs”. Now the model needs to know whether we are referring to Barack Obama or Michelle Obama so it needs to look back in the context to figure out which Obama which requires comprehension and world knowledge. For example if we precede with “After attending the national security briefing, …” then the model needs to know that: 1) national security briefings are attended by Presidents, 2) Barack Obama was President, in order to deduce that 3) “Obama” here is a “He”. Getting pronouns right matching human performances requires that model understands language and has some knowledge of the world.