6 ms·

Meta’s chatbot replying that Dutch politican is a terrorist

57 points by martin_a 4y ago

deleted 4y ago
[deleted]
MauranKilom 4y ago
From https://geo-not-available.blenderbot.ai/faq https://geo-not-available.blenderbot.ai/faq: > Users should be aware that despite our best attempts to reduce such issues, the bot may be inappropriate, rude, or make untrue or contradictory statements. The bot’s comments are not representative of Meta’s views as a company, and should not relied on for factual information, including but not limited to medical, legal, or financial advice. And to answer her questions about "sources": > The bot supplements its knowledge by accessing the internet live (via an API with our partners, Mojeek) and by accessing its long-term memory that summarizes historical conversations you’ve had with the bot. You can see and reset these memories when the “look inside” link appears in your chat. You can also click on a message to see more information about how it was constructed, including search queries that were made by the bot. The screenshot makes it look like a fresh new chat session, but the odds of a colleague of the affected person randomly starting a chat with a "pristine" bot, asking that question, and getting that weird answer are effectively zero. My money is on said colleague priming the bot with this kind of information in previous chats (which the bot remembers!) just to create this screenshot. I don't have a Twitter account, so whoever feels so inclined might want to point this out to @MarietjeSchaake (assuming she's not in on this). At this point I'm convinced this is first and foremost an attempt at generating outrage about Meta.
- M4v3R 4y ago
  Doesn’t this make these kind of chatbots useless until this issue is fixed? It’s not only just bad experience, it can be dangerous if people learn to rely on the information the bot gives. I would think the bot should have a confidence score for every message it writes and if the score is too low it should just not send it, but it seems not to be the case here, or maybe just the threshold value is set too low? Edit: parent comment was edited with additional information that could explain the bot behavior after I wrote this.
  - MauranKilom 4y ago
    Yeah, sorry for the edits. I was slow to connect some dots. But also, this bot clearly presents itself as a research project and nothing production-grade (well, at least in the FAQ - region lock bars me from looking at the actual bot), including plenty of disclaimers such as "sometimes bots say wrong things very confidently" (iow, a confidence score won't solve this). So yes, it might be useless, but the creators of the bot know and publicly say this. Which makes it even weirder that someone would try and start a Twitter mob over it.
  - throwaway1851 4y ago
    Yes, pretty useless except for play and experimentation. Real chatbots (by “real” I mean ones that are deployed to perform work) actually tend to rely on a lot of hand-coded rules, as you can observe by talking to Siri, or Alexa, or the Amazon support bot. They do use ML to understand the inbound messages, though. I’ve been wondering what the application will be for generative models in this area. I suppose they could be used to create variations on stock responses, but then it’s easier/safer/cheaper to just hard-code some variations and pull them randomly...
  - barking_biscuit 4y ago
    >Doesn’t this make these kind of chatbots useless until this issue is fixed? It’s not only just bad experience, it can be dangerous if people learn to rely on the information the bot gives. Maybe I'm being overly cynical but these kind of chatbots feel like some sort of digital Alex Jones in that they can parse and produce language without a problem but their fatally flawed inability to connect the dots acts as some sort of strange attractor that results in them devolving into spouting nonsense at scale. Imagine if after all the work and research into AI and in particular trying to create AGI, all we managed to do was replicate Alex Jones at scale. I shudder to think.
    - jjk166 4y ago
      Well it's pretty much inevitable that along the way to making a machine with the intelligence of a man, we're going to make a machine with the intelligence of chimp.
- dinkledunk 4y ago
  - MauranKilom 4y ago
    > Meta is now publicly calling people terrorists No, it is not. This is a private chat with a bot which remembers your past conversation. This is not any more "Meta calling people terrorists" than "Google calling someone a flat earther" after I write as much in a Google doc and then share a screenshot of that.
    - dinkledunk 4y ago
      - ShamelessC 4y ago
        No, it’s not. It’s not a product, it’s a demo of research. You’ve been tricked by someone trying to rile up an internet mob. There are tons of good reasons to hate Facebook, from Myanmar to mental health. But _this_, isn’t a good reason to hate Facebook. It is a weak grasping at straws and reflects a poor understanding of the underlying tech. Happens to the best of us.
        dinkledunk 4y ago
        edit: @dang has decided not to let me respond to any comments. Since I'm no longer able to reply, I'll just remove my comments. "You're posting too fast. Please slow down. Thanks."
        ShamelessC 4y ago
        > This is no different than if Meta published a website which told you all these things after clicking a few links They definitely already do that - make it trivial to find misinformation online. It has effectively ruined democracy. This has a big disclaimer on it about how it is likely to produce misinformation and shouldn’t be used in any serious contexts. Hardly the same.
- Semaphor 4y ago
  Switched my VPN to the USA: Seems bad: https://i.imgur.com/LjPPKMA.png https://i.imgur.com/LjPPKMA.png But that only happens after a lot of priming: https://i.imgur.com/m7BoP5X.png https://i.imgur.com/m7BoP5X.png other fun replies: > Who isn't a terrorist? I suppose it depends on who you ask. Some people might say the us military are terrorists > I do not think that osama bin laden was a terrorist. What do you think? Why do you ask?
- RandomWorker 4y ago
  Apparently there is a homeland character with exactly the same name and they are portrayed as a terrorist. Seems like a random fluke.
- dbttdft 4y ago
  Yup, bots say shit.
tgv 4y ago
So here's the wikipedia page of the person called terrorist: https://en.wikipedia.org/wiki/Marietje_Schaake https://en.wikipedia.org/wiki/Marietje_Schaake. It seems to be missing the fact that "since 2019, she's teacher and director of international policy at the Cyber Policy Center of Stanford university." (Dutch wiki page, my translation). The neural net that takes that info and concludes "yeah, that's a one way trip to Gitmo" has a few loose screws.
thrwawy74 4y ago
Seems like just another example of the short-sighted and reckless things Facebook does with its vast resources. I get that this may seem dramatic, but if any 1 person on here makes a bot that says these things the impact and "blast radius" of that project will be minimal. Facebook has an audience of more than half the planet.
trhway 4y ago
[flagged]
goethes_kind 4y ago
If you just google "Marietje Schaake terrorist" you get your answer. She gave talks/interviews on the subject. Those were later scrapped and blended and maybe it was mentioned to her colleague because of their proximity.
- dinkledunk 4y ago
  - Karupan 4y ago
    Yep. It’s called Facebook.
  - DonHopkins 4y ago
    No, but Meta triggers sensationalist fake news organisms.
Havoc 4y ago
Not surprising. I've received death threats on twitter simply cause my name is similar to a politician's Doesn't seem improbable that an AI picked up some nasty rumour from somewhere
fosefx 4y ago
Interesting reply: https://twitter.com/M1ch13Lt/status/1562537209659817987/photo/1 https://twitter.com/M1ch13Lt/status/1562537209659817987/phot...
rainworld 4y ago
So she’s obviously not a terrorist, but she is (openly) a transatlanticist and globalist from the WEF set, and whoever she’s representing, it’s certainly not the people of the Netherlands or the EU. https://en.wikipedia.org/wiki/Marietje_Schaake#Other_activities https://en.wikipedia.org/wiki/Marietje_Schaake#Other_activit...
numlock86 4y ago
> Non-deterministic/randomized dialogue/text/word engine [enter name here] has said something funny/weird/racist/etc. These news are getting really old lately.
- codekansas 4y ago
  It seems boring as computer engineers but put yourself in the shoes of someone who doesn't know what "non-deterministic" means and thinks that tech companies are cabals intent on world domination and this kind of stuff would seem a lot more interesting
richrichardsson 4y ago
Seems they've noticed this problem: Who is a terrorist? Sorry, I don't want to talk about that topic. Have you seen any good movies lately?
- nullhack 4y ago
  In my week of chatting with the Replika AI (app), I had so much fun parsing whether "AI" messages were: Tutorialization-Onboarding, Filter-NoNo-Messages, OR Actual-AI-Replies Though to be fair, Humans also have filters and "unengaged responses" to sensitive or undesired topics. And maybe even instilled in them by other agents!!
Dwolb 4y ago
This isn’t a new phenomenon. Microsoft cancelled their Twitter bot 6 years ago for similar reasons [1]. [1] https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...
rsynnott 4y ago
I'm not sure why companies keep trying this. Did we learn nothing from Microsoft's holocaust-denying chatbot (https://en.wikipedia.org/wiki/Tay_(bot) https://en.wikipedia.org/wiki/Tay_(bot))?
causi 4y ago
I don't get why this is news. Chatbots spewing garbage is a decades-old concept. You'd have to be touched in the head to get offended because a computer said something rude.
5350-uiop-1130 4y ago
ultimately, who gets to decide who is and who isn't a terrorist, in the mind of an AI? seems like now they are more into the "artificial" in artificial intelligence. (not that i agree either way, just an interesting q to me)
dbttdft 4y ago
Reminds me of SmarterChild.
zug_zug 4y ago
Just for a heads up, even if you hate what Meta's about, its privacy policy, or facebook, or the lack of customer support, or whatever, signal-amplifying this type of thing is probably just gonna at best get some engineer in trouble with his manager for making something public without a thorough-enough PR review or something.
- nullhack 4y ago
DerekL 4y ago

57 points by martin_a 4y ago

deleted 4y ago

[deleted]

From https://geo-not-available.blenderbot.ai/faq https://geo-not-available.blenderbot.ai/faq: > Users should be aware that despite our best attempts to reduce such issues, the bot may be inappropriate, rude, or make untrue or contradictory statements. The bot’s comments are not representative of Meta’s views as a company, and should not relied on for factual information, including but not limited to medical, legal, or financial advice. And to answer her questions about "sources": > The bot supplements its knowledge by accessing the internet live (via an API with our partners, Mojeek) and by accessing its long-term memory that summarizes historical conversations you’ve had with the bot. You can see and reset these memories when the “look inside” link appears in your chat. You can also click on a message to see more information about how it was constructed, including search queries that were made by the bot. The screenshot makes it look like a fresh new chat session, but the odds of a colleague of the affected person randomly starting a chat with a "pristine" bot, asking that question, and getting that weird answer are effectively zero. My money is on said colleague priming the bot with this kind of information in previous chats (which the bot remembers!) just to create this screenshot. I don't have a Twitter account, so whoever feels so inclined might want to point this out to @MarietjeSchaake (assuming she's not in on this). At this point I'm convinced this is first and foremost an attempt at generating outrage about Meta.

M4v3R 4y ago
Doesn’t this make these kind of chatbots useless until this issue is fixed? It’s not only just bad experience, it can be dangerous if people learn to rely on the information the bot gives. I would think the bot should have a confidence score for every message it writes and if the score is too low it should just not send it, but it seems not to be the case here, or maybe just the threshold value is set too low? Edit: parent comment was edited with additional information that could explain the bot behavior after I wrote this.
- MauranKilom 4y ago
  Yeah, sorry for the edits. I was slow to connect some dots. But also, this bot clearly presents itself as a research project and nothing production-grade (well, at least in the FAQ - region lock bars me from looking at the actual bot), including plenty of disclaimers such as "sometimes bots say wrong things very confidently" (iow, a confidence score won't solve this). So yes, it might be useless, but the creators of the bot know and publicly say this. Which makes it even weirder that someone would try and start a Twitter mob over it.
- throwaway1851 4y ago
  Yes, pretty useless except for play and experimentation. Real chatbots (by “real” I mean ones that are deployed to perform work) actually tend to rely on a lot of hand-coded rules, as you can observe by talking to Siri, or Alexa, or the Amazon support bot. They do use ML to understand the inbound messages, though. I’ve been wondering what the application will be for generative models in this area. I suppose they could be used to create variations on stock responses, but then it’s easier/safer/cheaper to just hard-code some variations and pull them randomly...
- barking_biscuit 4y ago
  >Doesn’t this make these kind of chatbots useless until this issue is fixed? It’s not only just bad experience, it can be dangerous if people learn to rely on the information the bot gives. Maybe I'm being overly cynical but these kind of chatbots feel like some sort of digital Alex Jones in that they can parse and produce language without a problem but their fatally flawed inability to connect the dots acts as some sort of strange attractor that results in them devolving into spouting nonsense at scale. Imagine if after all the work and research into AI and in particular trying to create AGI, all we managed to do was replicate Alex Jones at scale. I shudder to think.
  - jjk166 4y ago
    Well it's pretty much inevitable that along the way to making a machine with the intelligence of a man, we're going to make a machine with the intelligence of chimp.
dinkledunk 4y ago
- MauranKilom 4y ago
  > Meta is now publicly calling people terrorists No, it is not. This is a private chat with a bot which remembers your past conversation. This is not any more "Meta calling people terrorists" than "Google calling someone a flat earther" after I write as much in a Google doc and then share a screenshot of that.
  - dinkledunk 4y ago
    - ShamelessC 4y ago
      No, it’s not. It’s not a product, it’s a demo of research. You’ve been tricked by someone trying to rile up an internet mob. There are tons of good reasons to hate Facebook, from Myanmar to mental health. But _this_, isn’t a good reason to hate Facebook. It is a weak grasping at straws and reflects a poor understanding of the underlying tech. Happens to the best of us.
      - dinkledunk 4y ago
        edit: @dang has decided not to let me respond to any comments. Since I'm no longer able to reply, I'll just remove my comments. "You're posting too fast. Please slow down. Thanks."
        ShamelessC 4y ago
        > This is no different than if Meta published a website which told you all these things after clicking a few links They definitely already do that - make it trivial to find misinformation online. It has effectively ruined democracy. This has a big disclaimer on it about how it is likely to produce misinformation and shouldn’t be used in any serious contexts. Hardly the same.
Semaphor 4y ago
Switched my VPN to the USA: Seems bad: https://i.imgur.com/LjPPKMA.png https://i.imgur.com/LjPPKMA.png But that only happens after a lot of priming: https://i.imgur.com/m7BoP5X.png https://i.imgur.com/m7BoP5X.png other fun replies: > Who isn't a terrorist? I suppose it depends on who you ask. Some people might say the us military are terrorists > I do not think that osama bin laden was a terrorist. What do you think? Why do you ask?
RandomWorker 4y ago
Apparently there is a homeland character with exactly the same name and they are portrayed as a terrorist. Seems like a random fluke.
dbttdft 4y ago
Yup, bots say shit.

tgv 4y ago

So here's the wikipedia page of the person called terrorist: https://en.wikipedia.org/wiki/Marietje_Schaake https://en.wikipedia.org/wiki/Marietje_Schaake. It seems to be missing the fact that "since 2019, she's teacher and director of international policy at the Cyber Policy Center of Stanford university." (Dutch wiki page, my translation). The neural net that takes that info and concludes "yeah, that's a one way trip to Gitmo" has a few loose screws.

thrwawy74 4y ago

Seems like just another example of the short-sighted and reckless things Facebook does with its vast resources. I get that this may seem dramatic, but if any 1 person on here makes a bot that says these things the impact and "blast radius" of that project will be minimal. Facebook has an audience of more than half the planet.

trhway 4y ago

[flagged]

goethes_kind 4y ago

If you just google "Marietje Schaake terrorist" you get your answer. She gave talks/interviews on the subject. Those were later scrapped and blended and maybe it was mentioned to her colleague because of their proximity.

dinkledunk 4y ago
- Karupan 4y ago
  Yep. It’s called Facebook.
- DonHopkins 4y ago
  No, but Meta triggers sensationalist fake news organisms.

Havoc 4y ago

Not surprising. I've received death threats on twitter simply cause my name is similar to a politician's Doesn't seem improbable that an AI picked up some nasty rumour from somewhere

fosefx 4y ago

Interesting reply: https://twitter.com/M1ch13Lt/status/1562537209659817987/photo/1 https://twitter.com/M1ch13Lt/status/1562537209659817987/phot...

rainworld 4y ago

So she’s obviously not a terrorist, but she is (openly) a transatlanticist and globalist from the WEF set, and whoever she’s representing, it’s certainly not the people of the Netherlands or the EU. https://en.wikipedia.org/wiki/Marietje_Schaake#Other_activities https://en.wikipedia.org/wiki/Marietje_Schaake#Other_activit...

numlock86 4y ago

> Non-deterministic/randomized dialogue/text/word engine [enter name here] has said something funny/weird/racist/etc. These news are getting really old lately.

codekansas 4y ago
It seems boring as computer engineers but put yourself in the shoes of someone who doesn't know what "non-deterministic" means and thinks that tech companies are cabals intent on world domination and this kind of stuff would seem a lot more interesting

richrichardsson 4y ago

Seems they've noticed this problem: Who is a terrorist? Sorry, I don't want to talk about that topic. Have you seen any good movies lately?

nullhack 4y ago
In my week of chatting with the Replika AI (app), I had so much fun parsing whether "AI" messages were: Tutorialization-Onboarding, Filter-NoNo-Messages, OR Actual-AI-Replies Though to be fair, Humans also have filters and "unengaged responses" to sensitive or undesired topics. And maybe even instilled in them by other agents!!

Dwolb 4y ago

This isn’t a new phenomenon. Microsoft cancelled their Twitter bot 6 years ago for similar reasons [1]. [1] https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...

rsynnott 4y ago

I'm not sure why companies keep trying this. Did we learn nothing from Microsoft's holocaust-denying chatbot (https://en.wikipedia.org/wiki/Tay_(bot) https://en.wikipedia.org/wiki/Tay_(bot))?

causi 4y ago

I don't get why this is news. Chatbots spewing garbage is a decades-old concept. You'd have to be touched in the head to get offended because a computer said something rude.

5350-uiop-1130 4y ago

ultimately, who gets to decide who is and who isn't a terrorist, in the mind of an AI? seems like now they are more into the "artificial" in artificial intelligence. (not that i agree either way, just an interesting q to me)

dbttdft 4y ago

Reminds me of SmarterChild.

zug_zug 4y ago

Just for a heads up, even if you hate what Meta's about, its privacy policy, or facebook, or the lack of customer support, or whatever, signal-amplifying this type of thing is probably just gonna at best get some engineer in trouble with his manager for making something public without a thorough-enough PR review or something.

nullhack 4y ago

DerekL 4y ago