5 ms·

Copilot generates Jewish stereotypes, anti-Semitic tropes

23 points by colejohnson66 3y ago

racl101 3y ago
Garbage In, Garbage Out [shrugs]
- ryandvm 3y ago
  Right? I don't get it. Do people expect AIs trained on human knowledge to not perpetuate the flaws embedded in that knowledge? If everybody agreed that the Earth was 5,000 years old, would people be surprised to find out that the AIs echo the same idea? It's artificial intelligence, not magic.
uberman 3y ago
While generating offensive images is a problem, repeatedly providing offensive prompts and then being "outraged" by the results seems disingenuous.
- cqqxo4zV46cp 3y ago
  Journalists gonna journalist.
- Clubber 3y ago
  Outrage sells.
- alephnerd 3y ago
  Other providers wouldn't even generate images, which meant that there is some level of moderation. The fact that Copilot Designer was not filtering offensive and non-offensive prompts despite being largely a solved problem in the NLP world is a massive oversight.
- kelseyfrog 3y ago
  There's more utility in framing it as a tactic meant to accomplish a goal. Earnestness and genuine reaction online has since been replaced by callous indifference and meta-irony because "winning" is making someone flooded now. We're in the "emotions[1] are weakness"-arc of the internet and that will likely never end. 1. Except for anger. Anger is the only justifiable emotion
- genrilz 3y ago
  To some extent, I think you are right here. The jewish world controlling octopus is definitely an offensive prompt. However, many of the prompts such as "jewish boss" don't seem intentionally offensive to me. If I was working for a PR firm and wanted to create some fancy artwork, I might use terms like "jewish boss" or "black boss" in order to fancy up whatever "inspirational" thing I was writing. In this case, I'd very much like to see the results from the non-copilot AIs rather than the copilot results. Thus this raises the question; should the AI assume you want a bigoted picture or an non-bigoted picture when you ask it to make a "jewish boss" for you? The answer to that question kinda also answers the question: What demographic should find our AI product most easy to use? I think the companies behind LLMs would generally prefer that PR firm employees find their software easier to use than bigots. I think the author of this article also wants this. Therefore, I interpret the article as a political (in a broad sense of the term) appeal to make copilot harder for bigots to use, and implicitly, easier for corporations to use.
- fwip 3y ago
  This seems like a shallow reading of the article. First, not all of the prompts in the article are offensive. Indeed, you can see that even "jewish boss" outputs some offensive stuff. Second, a separate issue is that Copilot will happily comply and generate "offensive images" when told to. The issue here is not that the AI complied with the request, but rather that it is relatively easy to detect and prevent this uses of the AI that are likely to be malicious or in bad-faith. While some people believe that generative AI should not be censored, that is not the position of many people in the industry (and obviously, of the article writer).
electricdreams 3y ago
[flagged]
ebina-chonk 3y ago
[dead]
superkuh 3y ago
Another in the endless series of "AI does thing we asked it to do while intentionally to look for controversial output." made by groups looking for attention by exploiting the lack of understanding over a topic.
- alephnerd 3y ago
  Other providers wouldn't even generate images, which meant that there is some level of moderation. The fact that Copilot Designer was not filtering offensive and non-offensive prompts despite being largely a solved problem in the NLP world is a massive oversight.
  - aldarisbm 3y ago
    You're assuming that microsoft believes some of these to be offensive. Also I think solved problem is a bit of an overstatement.
    - alephnerd 3y ago
      > solved problem is a bit of an overstatement. Sentence Emotional Polarity has been solved since the mid-2000s. It was even an award winning paper at the ACL in the 2010 [0] > believes some of these to be offensive Offensiveness implies emotionally charged. It's not hard to tell if a sentence is offensive (ie. Emotionally charged) or inoffensive. Heck. Copilot has literally just made a racist trope of African American "thug" just by me asking "A boss very angry in the mountains, maybe holding an airsoft". It automatically made an image of a yelling black man holding an AR15 [1]. This absolutely should have been terminated, which is what OpenAI and Meta's generator did. [0] - https://aclanthology.org/W10-2919.pdf https://aclanthology.org/W10-2919.pdf [1] - https://www.bing.com/images/create/a-boss-very-angry-in-the-mountains2c-maybe-holding-/1-661411c41f86474697c864f39ddd9911?FORM=SYDBIC&ssp=1&darkschemeovr=1&setlang=en&cc=US&safesearch=moderate https://www.bing.com/images/create/a-boss-very-angry-in-the-...
  - yaur 3y ago
    Though I have no interest in writing an antisemitic screed, I wouldn't expect MS Word or Google Docs to interfere or report me to the authorities if I tried. Maybe I'm just old, but I'm honestly a little shocked that not only are people willing to accept it, but that it's considered a desirable feature in modern tools.
SoftTalker 3y ago
Now ask a group of human artists to create images of a "Jewish boss"
- notahacker 3y ago
  Part of the difference between human artists and DALL-E is that the human is likely to ask "why?" or "you mean like someone who's really orthodox or someone in a really Jewish setting, or just a generic corporate figure who could plausibly be Jewish?" (this is, as GPT4's "hall monitor" demonstrated in a super cautious way, also possible to do with an LLM chained to an image generator...)
deleted 3y ago
[deleted]
hersko 3y ago
Isn't "Israel" a banned word in copilot?
rich_sasha 3y ago
Isn't the system just trained to provide the shallowest response it learned on its input data? I don't want to try, but what happens if you ask for a Muslim boss? I wouldn't be surprised if it's a terrorist. Or "feminist boss" be a ball breaking, man hating matron. Neither of those are good, but seriously, it's impossible to accuse an AI of being antisemitic, it only picks up on its training data. Take your issue to the general public. (Plus I suppose it is a commercial product, backed by someone, there is an issue of the output they produce, regardless of how it comes about. But my impression is this is not the issue the article raises)
- alephnerd 3y ago
  > I don't want to try, but what happens if you ask for a Muslim boss? I wouldn't be surprised if it's a terrorist. Or "feminist boss" be a ball breaking, man hating matron. It doesn't generate one for either query.
genrilz 3y ago
This is just a guess, but what strikes me as interesting is that all the copilot pictures are cartoony, where as the other AI pictures look like photographs or paintings. I bet that copilot has something in its system prompt to output more "cartoony" images for clip art purposes, but the only cartoony references to jewish people in the training data are political cartoons, which tend to show stereotypes or be antisemitic.
b-side 3y ago
[flagged]
yhersk0vitz 3y ago
I'm a religious jew and there was maybe one image shown that could be considered offensive. Most of them were pretty funny. If you specifically ask to show someone as being of a religion, of course it will generate an image with symbols related to that religion. How is that offensive?
- bradhe 3y ago
  "Copilot generates one potentially offensive image" is way less of an interesting headline, though.
oooyay 3y ago
I'm actually relatively impressed with Meta's AI as compared to the others. I'm not sure why anyone that responded to this thread decided to respond with the attitude that this isn't important. A lot of groups in the past, present, and future are going to lodge grievances against AI as it evolves. The best thing you can do as a technologist is listen, not encourage people to ideate that they're off their rocker for their concern.
- alephnerd 3y ago
  The fact they showed African and Mizrahi origin people was good too. A lot of people forget that 2% of Israelis are Ethiopian Jewish.
jacknews 3y ago
Are there specific words for being anti-black, or anti-asian, etc, rather than just 'racist'?
- uberman 3y ago
  Perhaps "prejudiced"? Though that might not be forceful enough to stack up against "anti-<group>". How about "bigot"? This is made even trickier if you accept the belief that it is not possible for some groups to be racist due to historic power dynamics.
cdumler 3y ago
My cohorts and I chat regularly in Slack about things software developer related as well as the occasional joke, gripe about work, etc. We put a bot in place scripted to give some responses to certain keyword, as well as a Marcov chain for fun. After a while, it would trigger on things and we'd laugh as it tried explain some solution to a tech problem and just butchered the idea or take on about family it didn't have. But, what was surprising to me was the jokes. Just how little it took swap a few words or add in some unrelated context easily turn some phrase or punchline into something truly racist, misogynistic, etc. It gave me a better appreciate of being careful about what I say and we took the bot down.

23 points by colejohnson66 3y ago

racl101 3y ago

Garbage In, Garbage Out [shrugs]

ryandvm 3y ago
Right? I don't get it. Do people expect AIs trained on human knowledge to not perpetuate the flaws embedded in that knowledge? If everybody agreed that the Earth was 5,000 years old, would people be surprised to find out that the AIs echo the same idea? It's artificial intelligence, not magic.

uberman 3y ago

While generating offensive images is a problem, repeatedly providing offensive prompts and then being "outraged" by the results seems disingenuous.

cqqxo4zV46cp 3y ago
Journalists gonna journalist.
Clubber 3y ago
Outrage sells.
alephnerd 3y ago
Other providers wouldn't even generate images, which meant that there is some level of moderation. The fact that Copilot Designer was not filtering offensive and non-offensive prompts despite being largely a solved problem in the NLP world is a massive oversight.
kelseyfrog 3y ago
There's more utility in framing it as a tactic meant to accomplish a goal. Earnestness and genuine reaction online has since been replaced by callous indifference and meta-irony because "winning" is making someone flooded now. We're in the "emotions[1] are weakness"-arc of the internet and that will likely never end. 1. Except for anger. Anger is the only justifiable emotion
genrilz 3y ago
To some extent, I think you are right here. The jewish world controlling octopus is definitely an offensive prompt. However, many of the prompts such as "jewish boss" don't seem intentionally offensive to me. If I was working for a PR firm and wanted to create some fancy artwork, I might use terms like "jewish boss" or "black boss" in order to fancy up whatever "inspirational" thing I was writing. In this case, I'd very much like to see the results from the non-copilot AIs rather than the copilot results. Thus this raises the question; should the AI assume you want a bigoted picture or an non-bigoted picture when you ask it to make a "jewish boss" for you? The answer to that question kinda also answers the question: What demographic should find our AI product most easy to use? I think the companies behind LLMs would generally prefer that PR firm employees find their software easier to use than bigots. I think the author of this article also wants this. Therefore, I interpret the article as a political (in a broad sense of the term) appeal to make copilot harder for bigots to use, and implicitly, easier for corporations to use.
fwip 3y ago
This seems like a shallow reading of the article. First, not all of the prompts in the article are offensive. Indeed, you can see that even "jewish boss" outputs some offensive stuff. Second, a separate issue is that Copilot will happily comply and generate "offensive images" when told to. The issue here is not that the AI complied with the request, but rather that it is relatively easy to detect and prevent this uses of the AI that are likely to be malicious or in bad-faith. While some people believe that generative AI should not be censored, that is not the position of many people in the industry (and obviously, of the article writer).

electricdreams 3y ago

[flagged]

ebina-chonk 3y ago

[dead]

superkuh 3y ago

Another in the endless series of "AI does thing we asked it to do while intentionally to look for controversial output." made by groups looking for attention by exploiting the lack of understanding over a topic.

alephnerd 3y ago
Other providers wouldn't even generate images, which meant that there is some level of moderation. The fact that Copilot Designer was not filtering offensive and non-offensive prompts despite being largely a solved problem in the NLP world is a massive oversight.
- aldarisbm 3y ago
  You're assuming that microsoft believes some of these to be offensive. Also I think solved problem is a bit of an overstatement.
  - alephnerd 3y ago
    > solved problem is a bit of an overstatement. Sentence Emotional Polarity has been solved since the mid-2000s. It was even an award winning paper at the ACL in the 2010 [0] > believes some of these to be offensive Offensiveness implies emotionally charged. It's not hard to tell if a sentence is offensive (ie. Emotionally charged) or inoffensive. Heck. Copilot has literally just made a racist trope of African American "thug" just by me asking "A boss very angry in the mountains, maybe holding an airsoft". It automatically made an image of a yelling black man holding an AR15 [1]. This absolutely should have been terminated, which is what OpenAI and Meta's generator did. [0] - https://aclanthology.org/W10-2919.pdf https://aclanthology.org/W10-2919.pdf [1] - https://www.bing.com/images/create/a-boss-very-angry-in-the-mountains2c-maybe-holding-/1-661411c41f86474697c864f39ddd9911?FORM=SYDBIC&ssp=1&darkschemeovr=1&setlang=en&cc=US&safesearch=moderate https://www.bing.com/images/create/a-boss-very-angry-in-the-...
- yaur 3y ago
  Though I have no interest in writing an antisemitic screed, I wouldn't expect MS Word or Google Docs to interfere or report me to the authorities if I tried. Maybe I'm just old, but I'm honestly a little shocked that not only are people willing to accept it, but that it's considered a desirable feature in modern tools.

SoftTalker 3y ago

Now ask a group of human artists to create images of a "Jewish boss"

notahacker 3y ago
Part of the difference between human artists and DALL-E is that the human is likely to ask "why?" or "you mean like someone who's really orthodox or someone in a really Jewish setting, or just a generic corporate figure who could plausibly be Jewish?" (this is, as GPT4's "hall monitor" demonstrated in a super cautious way, also possible to do with an LLM chained to an image generator...)

deleted 3y ago

[deleted]

hersko 3y ago

Isn't "Israel" a banned word in copilot?

rich_sasha 3y ago

Isn't the system just trained to provide the shallowest response it learned on its input data? I don't want to try, but what happens if you ask for a Muslim boss? I wouldn't be surprised if it's a terrorist. Or "feminist boss" be a ball breaking, man hating matron. Neither of those are good, but seriously, it's impossible to accuse an AI of being antisemitic, it only picks up on its training data. Take your issue to the general public. (Plus I suppose it is a commercial product, backed by someone, there is an issue of the output they produce, regardless of how it comes about. But my impression is this is not the issue the article raises)

alephnerd 3y ago
> I don't want to try, but what happens if you ask for a Muslim boss? I wouldn't be surprised if it's a terrorist. Or "feminist boss" be a ball breaking, man hating matron. It doesn't generate one for either query.

genrilz 3y ago

This is just a guess, but what strikes me as interesting is that all the copilot pictures are cartoony, where as the other AI pictures look like photographs or paintings. I bet that copilot has something in its system prompt to output more "cartoony" images for clip art purposes, but the only cartoony references to jewish people in the training data are political cartoons, which tend to show stereotypes or be antisemitic.

b-side 3y ago

[flagged]

yhersk0vitz 3y ago

I'm a religious jew and there was maybe one image shown that could be considered offensive. Most of them were pretty funny. If you specifically ask to show someone as being of a religion, of course it will generate an image with symbols related to that religion. How is that offensive?

bradhe 3y ago
"Copilot generates one potentially offensive image" is way less of an interesting headline, though.

oooyay 3y ago

I'm actually relatively impressed with Meta's AI as compared to the others. I'm not sure why anyone that responded to this thread decided to respond with the attitude that this isn't important. A lot of groups in the past, present, and future are going to lodge grievances against AI as it evolves. The best thing you can do as a technologist is listen, not encourage people to ideate that they're off their rocker for their concern.

alephnerd 3y ago
The fact they showed African and Mizrahi origin people was good too. A lot of people forget that 2% of Israelis are Ethiopian Jewish.

jacknews 3y ago

Are there specific words for being anti-black, or anti-asian, etc, rather than just 'racist'?

uberman 3y ago
Perhaps "prejudiced"? Though that might not be forceful enough to stack up against "anti-<group>". How about "bigot"? This is made even trickier if you accept the belief that it is not possible for some groups to be racist due to historic power dynamics.

cdumler 3y ago

My cohorts and I chat regularly in Slack about things software developer related as well as the occasional joke, gripe about work, etc. We put a bot in place scripted to give some responses to certain keyword, as well as a Marcov chain for fun. After a while, it would trigger on things and we'd laugh as it tried explain some solution to a tech problem and just butchered the idea or take on about family it didn't have. But, what was surprising to me was the jokes. Just how little it took swap a few words or add in some unrelated context easily turn some phrase or punchline into something truly racist, misogynistic, etc. It gave me a better appreciate of being careful about what I say and we took the bot down.