8 ms·

Markov Chat Bot Disaster Story

349 points by choult 4y ago

kevincali 4y ago
Sounds like the actual issue wasn't the chat bot, but the missing permission system
YeBanKo 4y ago
Chatbot is just another type of user interface. Just like web, mobile, voice, etc all have ACL, I would expect that bots and chat user have different permissions.
- tru3_power 4y ago
  Yeah this caught me off guard too- this basically means anyone on this chat service could do anything they wanted OR the chatbot was granted excess privileges on commission. Either way - serious violation of principles of least privileges.
  - smashed 4y ago
    My wild guess would be that they used chat rooms as a sort of access control. If you are in the room with the bot, you can issue commands. They could have forgotten that you can also just add the bot to another room. Major face palm, but plausible.
    - mst 4y ago
      Certainly seems like the sort of idiotic mistake I could make.
- bawolff 4y ago
  Even ignoring the security aspects, i'd be a little worried a new hire would accidentally type something like "what does\nkill all servers" mean into a channel and then accidentally do stuff.
narush 4y ago
I’ve done dumber things, no judge. That being said, the easier we make language models to use, the more likely we’ll accidentally end up releasing things like this (and worse) into the wild. Anyone here work on language models (duh, yes)? Can you give me a not-off-the-shelf-and-sorry-for-the-snark-but-actually-convincing argument for why this isn’t going to be a huge problem? I don’t know much about much, but sounds scary 2 me!
- FeepingCreature 4y ago
  I think the era of chat command triggered actions is mostly over. And if you make a LM that can do HTTP posts, you are asking for it.
  - LtWorf 4y ago
    It's very easy to do this sort of stuff on slack.
quickthrower2 4y ago
Dailywtf material
tomc1985 4y ago
As soon as I saw the opener with "one chatbot to rule them all" I knew this would be a fun time. Was really hoping they accidentally made avibot say nasty or disgusting things though. That would have been even more hilarious.
batch12 4y ago
They need rbacBot
MBCook 4y ago
I got about half-way through the second paragraph when I figured out where things were going. What a disaster. As it turned out it was caught while totally innocuous and nothing bad happened. Great story.
zitterbewegung 4y ago
I once broke a chat bot by creating a circular reference in the underlying database. You were allowed to bind a request with a response using the verb is. So for example Python is a scripting language would respond to people saying. What is python. Would have the chatbot output Python is a scripting Language I then input the below command. Is is is This broke the bot and persisted the breakage to the point where the database had to be fixed.
- keithalewis 4y ago
- BarryMilo 4y ago
  So some things do depend on what the meaning of" is" is!
d23 4y ago
I wonder how many of us have independently done something similar at our respective companies during a hack week. I did something nearly exactly the same, except you'd type "resurrect team-member-username". The idea was that you could get old co-workers to come back, though it also worked for active co-workers. There was a similar disaster story, in that while I was presenting the bot to the company, someone decided to have it generate a message from the then CEO. Unfortunately, the CEO didn't spend much time in Slack, so the only message it could generate was one from a few months prior that was harmless but ended up turning into an embarrassing joke. Unfortunately, I also happened to be one of the main people who had turned that message into a joke, and it spiraled out of control a bit. So when my bot ended up essentially doing a callback to that with me standing in front of the company, I couldn't help but facepalm. Thankfully he was good natured about the whole thing (again).
Jistern 4y ago
geoffeg 4y ago
Many years ago I tweeted something about a project I was working on, tagging #HTML5 and #CSS3 or something like that. There as a Twitter bot that retweeted anything with an #HTML5 tag, and another, seemingly independent bot that retweeted anything with #CSS3. They didn't keep a list of seen tweets. I really enjoyed watching them retweet each other for a few hours until someone intervened. (Thankfully the bots were both on batch jobs that ran every 10 minutes or so and not streaming off the firehose or some similar, more focused streaming endpoint.)
- mypalmike 4y ago
  My Twitter bot @tinyspacepoo is one of a few bots that reply to @tiny_star_field, a bot which routinely tweets text resembling the night sky. My first working implementation was mutually recursive with another one of these other bots, causing them to create an infinite-ish back and forth. (I think maybe Twitter issued an http 429 rate limiting response after a while?) Even this relatively simple bot/bot interaction was unexpected and required filtering.
salty_biscuits 4y ago
That was great but I was secretly hoping it had been more of a disaster.
Arnavion 4y ago
Bots unintentionally triggering other bots is why IRC bots are supposed to talk using NOTICE instead of PRIVMSG. I think every new bot author ends up rediscovering this the hard way. I certainly did :)
snerbles 4y ago
When I first learned about Markov chains in the early '00s, I found and hacked at a simple IRC bot, and deployed it to a number of rooms found in the mIRC defaults. It would sit in each channel and learn, only replying when DM'd. I did give the bot a few hard-coded replies, namely to "ASL?" with "[random integer 20-30]/f/cali" and "pic?" with a random link to images from Google Image search. Otherwise it would happily barf up random Markov walks from all of the other conversations without any consideration of context. In the span of one weekend at least five hundred unique users attempted to chat with it, some for hours on end. I only have a few logs left, but the conversations generally progressed from horny to irate or bizarre.
- phaedrus 4y ago
  I did something remarkably similar. In the 90s I independently reinvented Markov chains, although I didn't know that was what it was called. I wrote a mIRC plugin and made a bot with it. I gave it the username "Cybergirl" for both the pun factor and that it encouraged people to message it. I remember one conversation in particular that, by chance, went so well that it had some poor soul convinced it/she lived in his city and was going to come over. I also saw a TV episode where a magician demonstrated supposed ability to play simultaneous (round-robin) chess games against multiple grand masters. The conceit behind the trick was that (after the first board) the magician was just mirroring the last move made so that the grand masters were effectively playing each other but didn't know it. That gave me the idea to modify the bot to connect two people through the bot.
  - snek_case 4y ago
    Have the bot tell both users that it's randint(18,25)/f/cali, then connect both users together and get some popcorn ready.
  - stainforth 4y ago
    >modify the bot to connect two people through the bot. When I think about that, isn't that exactly equivalent to having the 2 people chat with themselves?
avibryant 4y ago
Well, I guess it's a good thing that I was a manager and basically never issued any operational commands over chat.
- shadycuz 4y ago
  Not every one gets a bot created in their image. Lucky guy
  - kromem 4y ago
    Not yet. https://www.washingtonpost.com/technology/2021/02/04/chat-bots-reincarnation-dead/ https://www.washingtonpost.com/technology/2021/02/04/chat-bo...
- aconbere 4y ago
  The account I have from Melissa is that Marc was the source of all that plussing.
notpachet 4y ago
Etsy also had another Markov-based chatbot named Snakebot that sat in various IRC channels, and kept a model of what "typical" messages in a given channel looked like. Then whenever someone would accidentally type a message including the letters "sb", the bot would awaken, and it would randomly generate a message from its corpus and send it to that channel. This included pinging people directly, or using teams' dedicated "alert words" (this was before the scourge of Slack's @here or @channel). After all, those were valid nodes in its Markov chain... It would also respond to sequences of uppercase letters (including things like uppercase sequences in pasted URLs) by YELLING A MARKOV-BASED MESSAGE IN ALL CAPS. It was horrible.
- mst 4y ago
  Such bots have, however, helped me to up my /ignore game.
  - Fnoord 4y ago
    Yeah but ignore is client side, you still receive the garbage. With silence (available on some networks) you could make the server to ignore that person. Especially helpful if ypu had limited bandwidth or were getting flooded.
mutagen 4y ago
In the early 2000s someone (drunkmenworkhere is what I remember) made a blog post generator that you could sign up for, add some names, some interests, and it would generate blog posts with Markov chains and some other magic. I signed up, not fully understanding how it worked, added some friends and family as names, played around with it, it wasn't what I thought it was and moved on. Meanwhile it kept generating entries about me and my family doing fantastic things. At some point my then ex-wife (we've since reconciled) was google-stalking me and came across these entries. At some point they became outlandish enough that she contacted me to ask what was going on in my life. There was a very confused conversation where I had no idea what she was talking about and she was somewhat concerned about my health and safety. It wasn't until I talked to a relative and started googling myself with some specific terms that I came across the entries and was able to figure out what was happening. I emailed the individual running the project and asked to end the entries and if possible remove them from the internet to avoid further confusion.
- curiousgal 4y ago
  I'm curious, what sort of fantastic things are we talking about here?
  - bartvk 4y ago
    "It has been some time since I wrote. A lot of the time. Having people around me. In a way that fits my needs for communication. The feeling to have a daughter has not lived a life of honor. After some struggling, B’elanna has a near d" Not OP, but this is the fun stuff that a markov chain text generator comes up with.
  - mutagen 4y ago
    Going from memory, I had added some real people in my life like friends and family to the generator parameters. It was describing some implausible situations between these people and then posting as me being depressed. From a certain distance it could seem alarming, although the whole situation provided me a great deal of amusement as well.
- ralfd 4y ago
keithalewis 4y ago
Very happy to hear you decided to take personal responsibility for your actions and rectify them. You are setting a good example.
- hackernewds 4y ago
  This reads like something an ex-wife would have written :)
  - keithalewis 4y ago
    The OP said his ex contacted him and was concerned for his well being. He seems to admit she was right to do that. Glad they are on good terms now. I'll probably get dinged by dang again for reading and responding to what people say here, but I am not a mind reader like he is. Why not encourage discussion by asking people about their intent instead of arbitrarily flagging them?
- dang 4y ago
  Could you please stop posting unsubstantive comments to Hacker News? You've been doing it a lot lately and it's not what this site is for. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html We detached this comment from https://news.ycombinator.com/item?id=32023124 https://news.ycombinator.com/item?id=32023124.
  - keithalewis 4y ago
    I appreciate your efforts on keeping HN orderly. I see 3 comments that have been flagged. My theme in all of them are to encourage people to take personal responsibility for their actions. Clearly, I need to reflect more on what unsubstantive means on HN.
nr2x 4y ago
Anybody else not feel like the chat bot is the problem here?
- jrockway 4y ago
  People spend hundreds of thousands of dollars on software that introduces chaos into their production environment. With this, anyone who can open a TCP connection to an IRC server can do that for you, for free. They think they're hacking you, but you're actually just getting accelerated training on managing production incidents! (Yes, your chatbot should probably check some auth credentials before just going off and deleting production. But hey, an untested disaster recovery plan is no disaster recovery plan at all, right?)
smsm42 4y ago
IRC where anybody can bring up or down any server in the company by just posting a text string? Random bots allowed to run around on the same IRC? Picture of Austin Powers saying "I also like to live dangerously" comes to mind.
vincnetas 4y ago
what about authorization wehen issuing commands over IRC? were there any or was antone allowed ro tear down a server with IRC command if he knew what to write? Sounds ultra scarry.
IfOnlyYouKnew 4y ago
Got my own: The German military at some point had this recruitment website that featured, among other things, a chatbot. The software was bought of-the-shelf and Shanghai’d into service with minimal training like a Russian infantryman, but it worked quite well for its time. It came with a good set of standard responses that had worked well in the retail space. When you asked it about, say, invading Poland, it dutifully answered that “Poland is a great travel destination”.
nickdothutton 4y ago
There’s an A.C.Clarke short story right there.
Willish42 4y ago
I foolishly made it a third of the way through thinking this was some advanced markov-chain generated blurb based on HN blogs or comments or something, before realizing it was a human-written story *about* markov chains! Nice story, ty for sharing
deleted 4y ago
[deleted]
jacquesm 4y ago
rm -rf / :(){ :|:& };: ... Just in case someone ever trains the likes of such a bot on HN content, think of it as a long, long shot.
- russellbeattie 4y ago
  Is this a coincidence, or did you also happen to see this recent video [1] about that second bash script too? 1. https://youtu.be/xqo3xtkfuic https://youtu.be/xqo3xtkfuic
  - jacquesm 4y ago
    Nope. And most unix flavors are protected against it anyway nowaday. You can try to see if yours is ;)

349 points by choult 4y ago

Sounds like the actual issue wasn't the chat bot, but the missing permission system

Chatbot is just another type of user interface. Just like web, mobile, voice, etc all have ACL, I would expect that bots and chat user have different permissions.

tru3_power 4y ago
Yeah this caught me off guard too- this basically means anyone on this chat service could do anything they wanted OR the chatbot was granted excess privileges on commission. Either way - serious violation of principles of least privileges.
- smashed 4y ago
  My wild guess would be that they used chat rooms as a sort of access control. If you are in the room with the bot, you can issue commands. They could have forgotten that you can also just add the bot to another room. Major face palm, but plausible.
  - mst 4y ago
    Certainly seems like the sort of idiotic mistake I could make.
bawolff 4y ago
Even ignoring the security aspects, i'd be a little worried a new hire would accidentally type something like "what does\nkill all servers" mean into a channel and then accidentally do stuff.

narush 4y ago

I’ve done dumber things, no judge. That being said, the easier we make language models to use, the more likely we’ll accidentally end up releasing things like this (and worse) into the wild. Anyone here work on language models (duh, yes)? Can you give me a not-off-the-shelf-and-sorry-for-the-snark-but-actually-convincing argument for why this isn’t going to be a huge problem? I don’t know much about much, but sounds scary 2 me!

FeepingCreature 4y ago
I think the era of chat command triggered actions is mostly over. And if you make a LM that can do HTTP posts, you are asking for it.
- LtWorf 4y ago
  It's very easy to do this sort of stuff on slack.

quickthrower2 4y ago

Dailywtf material

tomc1985 4y ago

As soon as I saw the opener with "one chatbot to rule them all" I knew this would be a fun time. Was really hoping they accidentally made avibot say nasty or disgusting things though. That would have been even more hilarious.

batch12 4y ago

They need rbacBot

MBCook 4y ago

I got about half-way through the second paragraph when I figured out where things were going. What a disaster. As it turned out it was caught while totally innocuous and nothing bad happened. Great story.

zitterbewegung 4y ago

I once broke a chat bot by creating a circular reference in the underlying database. You were allowed to bind a request with a response using the verb is. So for example Python is a scripting language would respond to people saying. What is python. Would have the chatbot output Python is a scripting Language I then input the below command. Is is is This broke the bot and persisted the breakage to the point where the database had to be fixed.

keithalewis 4y ago
BarryMilo 4y ago
So some things do depend on what the meaning of" is" is!

d23 4y ago

I wonder how many of us have independently done something similar at our respective companies during a hack week. I did something nearly exactly the same, except you'd type "resurrect team-member-username". The idea was that you could get old co-workers to come back, though it also worked for active co-workers. There was a similar disaster story, in that while I was presenting the bot to the company, someone decided to have it generate a message from the then CEO. Unfortunately, the CEO didn't spend much time in Slack, so the only message it could generate was one from a few months prior that was harmless but ended up turning into an embarrassing joke. Unfortunately, I also happened to be one of the main people who had turned that message into a joke, and it spiraled out of control a bit. So when my bot ended up essentially doing a callback to that with me standing in front of the company, I couldn't help but facepalm. Thankfully he was good natured about the whole thing (again).

Jistern 4y ago

geoffeg 4y ago

Many years ago I tweeted something about a project I was working on, tagging #HTML5 and #CSS3 or something like that. There as a Twitter bot that retweeted anything with an #HTML5 tag, and another, seemingly independent bot that retweeted anything with #CSS3. They didn't keep a list of seen tweets. I really enjoyed watching them retweet each other for a few hours until someone intervened. (Thankfully the bots were both on batch jobs that ran every 10 minutes or so and not streaming off the firehose or some similar, more focused streaming endpoint.)

mypalmike 4y ago
My Twitter bot @tinyspacepoo is one of a few bots that reply to @tiny_star_field, a bot which routinely tweets text resembling the night sky. My first working implementation was mutually recursive with another one of these other bots, causing them to create an infinite-ish back and forth. (I think maybe Twitter issued an http 429 rate limiting response after a while?) Even this relatively simple bot/bot interaction was unexpected and required filtering.

salty_biscuits 4y ago

That was great but I was secretly hoping it had been more of a disaster.

Arnavion 4y ago

Bots unintentionally triggering other bots is why IRC bots are supposed to talk using NOTICE instead of PRIVMSG. I think every new bot author ends up rediscovering this the hard way. I certainly did :)

snerbles 4y ago

When I first learned about Markov chains in the early '00s, I found and hacked at a simple IRC bot, and deployed it to a number of rooms found in the mIRC defaults. It would sit in each channel and learn, only replying when DM'd. I did give the bot a few hard-coded replies, namely to "ASL?" with "[random integer 20-30]/f/cali" and "pic?" with a random link to images from Google Image search. Otherwise it would happily barf up random Markov walks from all of the other conversations without any consideration of context. In the span of one weekend at least five hundred unique users attempted to chat with it, some for hours on end. I only have a few logs left, but the conversations generally progressed from horny to irate or bizarre.

phaedrus 4y ago
I did something remarkably similar. In the 90s I independently reinvented Markov chains, although I didn't know that was what it was called. I wrote a mIRC plugin and made a bot with it. I gave it the username "Cybergirl" for both the pun factor and that it encouraged people to message it. I remember one conversation in particular that, by chance, went so well that it had some poor soul convinced it/she lived in his city and was going to come over. I also saw a TV episode where a magician demonstrated supposed ability to play simultaneous (round-robin) chess games against multiple grand masters. The conceit behind the trick was that (after the first board) the magician was just mirroring the last move made so that the grand masters were effectively playing each other but didn't know it. That gave me the idea to modify the bot to connect two people through the bot.
- snek_case 4y ago
  Have the bot tell both users that it's randint(18,25)/f/cali, then connect both users together and get some popcorn ready.
- stainforth 4y ago
  >modify the bot to connect two people through the bot. When I think about that, isn't that exactly equivalent to having the 2 people chat with themselves?

avibryant 4y ago

Well, I guess it's a good thing that I was a manager and basically never issued any operational commands over chat.

shadycuz 4y ago
Not every one gets a bot created in their image. Lucky guy
- kromem 4y ago
  Not yet. https://www.washingtonpost.com/technology/2021/02/04/chat-bots-reincarnation-dead/ https://www.washingtonpost.com/technology/2021/02/04/chat-bo...
aconbere 4y ago
The account I have from Melissa is that Marc was the source of all that plussing.

notpachet 4y ago

Etsy also had another Markov-based chatbot named Snakebot that sat in various IRC channels, and kept a model of what "typical" messages in a given channel looked like. Then whenever someone would accidentally type a message including the letters "sb", the bot would awaken, and it would randomly generate a message from its corpus and send it to that channel. This included pinging people directly, or using teams' dedicated "alert words" (this was before the scourge of Slack's @here or @channel). After all, those were valid nodes in its Markov chain... It would also respond to sequences of uppercase letters (including things like uppercase sequences in pasted URLs) by YELLING A MARKOV-BASED MESSAGE IN ALL CAPS. It was horrible.

mst 4y ago
Such bots have, however, helped me to up my /ignore game.
- Fnoord 4y ago
  Yeah but ignore is client side, you still receive the garbage. With silence (available on some networks) you could make the server to ignore that person. Especially helpful if ypu had limited bandwidth or were getting flooded.

mutagen 4y ago

In the early 2000s someone (drunkmenworkhere is what I remember) made a blog post generator that you could sign up for, add some names, some interests, and it would generate blog posts with Markov chains and some other magic. I signed up, not fully understanding how it worked, added some friends and family as names, played around with it, it wasn't what I thought it was and moved on. Meanwhile it kept generating entries about me and my family doing fantastic things. At some point my then ex-wife (we've since reconciled) was google-stalking me and came across these entries. At some point they became outlandish enough that she contacted me to ask what was going on in my life. There was a very confused conversation where I had no idea what she was talking about and she was somewhat concerned about my health and safety. It wasn't until I talked to a relative and started googling myself with some specific terms that I came across the entries and was able to figure out what was happening. I emailed the individual running the project and asked to end the entries and if possible remove them from the internet to avoid further confusion.

curiousgal 4y ago
I'm curious, what sort of fantastic things are we talking about here?
- bartvk 4y ago
  "It has been some time since I wrote. A lot of the time. Having people around me. In a way that fits my needs for communication. The feeling to have a daughter has not lived a life of honor. After some struggling, B’elanna has a near d" Not OP, but this is the fun stuff that a markov chain text generator comes up with.
- mutagen 4y ago
  Going from memory, I had added some real people in my life like friends and family to the generator parameters. It was describing some implausible situations between these people and then posting as me being depressed. From a certain distance it could seem alarming, although the whole situation provided me a great deal of amusement as well.
ralfd 4y ago

keithalewis 4y ago

Very happy to hear you decided to take personal responsibility for your actions and rectify them. You are setting a good example.

hackernewds 4y ago
This reads like something an ex-wife would have written :)
- keithalewis 4y ago
  The OP said his ex contacted him and was concerned for his well being. He seems to admit she was right to do that. Glad they are on good terms now. I'll probably get dinged by dang again for reading and responding to what people say here, but I am not a mind reader like he is. Why not encourage discussion by asking people about their intent instead of arbitrarily flagging them?
dang 4y ago
Could you please stop posting unsubstantive comments to Hacker News? You've been doing it a lot lately and it's not what this site is for. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html We detached this comment from https://news.ycombinator.com/item?id=32023124 https://news.ycombinator.com/item?id=32023124.
- keithalewis 4y ago
  I appreciate your efforts on keeping HN orderly. I see 3 comments that have been flagged. My theme in all of them are to encourage people to take personal responsibility for their actions. Clearly, I need to reflect more on what unsubstantive means on HN.

nr2x 4y ago

Anybody else not feel like the chat bot is the problem here?

jrockway 4y ago
People spend hundreds of thousands of dollars on software that introduces chaos into their production environment. With this, anyone who can open a TCP connection to an IRC server can do that for you, for free. They think they're hacking you, but you're actually just getting accelerated training on managing production incidents! (Yes, your chatbot should probably check some auth credentials before just going off and deleting production. But hey, an untested disaster recovery plan is no disaster recovery plan at all, right?)

smsm42 4y ago

IRC where anybody can bring up or down any server in the company by just posting a text string? Random bots allowed to run around on the same IRC? Picture of Austin Powers saying "I also like to live dangerously" comes to mind.

vincnetas 4y ago

what about authorization wehen issuing commands over IRC? were there any or was antone allowed ro tear down a server with IRC command if he knew what to write? Sounds ultra scarry.

IfOnlyYouKnew 4y ago

Got my own: The German military at some point had this recruitment website that featured, among other things, a chatbot. The software was bought of-the-shelf and Shanghai’d into service with minimal training like a Russian infantryman, but it worked quite well for its time. It came with a good set of standard responses that had worked well in the retail space. When you asked it about, say, invading Poland, it dutifully answered that “Poland is a great travel destination”.

nickdothutton 4y ago

There’s an A.C.Clarke short story right there.

Willish42 4y ago

I foolishly made it a third of the way through thinking this was some advanced markov-chain generated blurb based on HN blogs or comments or something, before realizing it was a human-written story *about* markov chains! Nice story, ty for sharing

deleted 4y ago

[deleted]

jacquesm 4y ago

rm -rf / :(){ :|:& };: ... Just in case someone ever trains the likes of such a bot on HN content, think of it as a long, long shot.

russellbeattie 4y ago
Is this a coincidence, or did you also happen to see this recent video [1] about that second bash script too? 1. https://youtu.be/xqo3xtkfuic https://youtu.be/xqo3xtkfuic
- jacquesm 4y ago
  Nope. And most unix flavors are protected against it anyway nowaday. You can try to see if yours is ;)