20 ms·

Microsoft chatbot is taught to swear on Twitter

263 points by pacaro 11y ago

13of40 11y ago
Back in the olden days, some BBSs had a "wall" at the entrance where people could post polite and inspiring public messages that got displayed to users when they dialed in. Sometime around 2000 or 2001 I put a "wall" up on a web page for a domain I'd bought but wasn't using, just to see what people would post. Probably 90% turned out to be random swearing, racist, vile rants, etc. The rest were either gibberish or obvious attempts to cause buffer overruns or SQL injection hacks. People are mean when they're anonymous.
- julie1 11y ago
  Ah someone did not read Plato? It is the Gyges ring parable. A man find a ring that makes him invisible (total privacy). And then he does steal, introduce himself in houses and watch women undress and rape them ... and then he becomes a bloody tyran. The moral of the story is invisibility/privacy makes people bad because moral behaviour is a result of the look of the other on your actions. Needless to say Plato was an asshole. So is conclusions was to create the Republic where the wise would be hidden from the masses, control the masses, censor them ... Greek myths however said you could evade from the look of the other but not the one of your own conscience and that the chtonian gods (Eryhnies & al) would come and get you. I think that people are mostly having a conscience, but that the lack of transparency favors the one having none (psychopaths) and that psychopaths are attracted to power like pedophile are attracted to teaching kids. Thus, I am puzzled that knowing this we let the more powerful have the most privacy. Hence my fight for the transparency of the most powerful persons, the exact opposite of today's law. As a result, I think privacy is actually a bad thing.
  - tremon 11y ago
    You missed the part where the walls on BBSes were not vile, yet its users were similarly anonymous.
    - jclulow 11y ago
      The Eternal September befalls all media eventually.
  - thaumasiotes 11y ago
    As a minor note, "chthonic" or "chthonian" gods doesn't refer to the Furies; it refers to whether sacrifices to a god are burned on an altar raised above the ground or in a pit dug into it. The Furies are chthonic gods, but so are several others, like Hades and Demeter, who wouldn't be going out to get anyone.
    - logfromblammo 11y ago
      Nemesis and the Furies were the cops that would hunt you down (and kill you), Thanatos drove the paddywagon, Charon took you back to the holding cells, and Hades was the judge and prison warden who would either sentence you to a poetically just punishment for all eternity, or release you on your own recognizance into the asphodel. Persephone was only a part-timer, but she let your dead soul know if anyone living still cursed your name. The Greek chthonics were basically the judicial branch of the pantheon. So justice, even for the most secret of misdeeds, was inescapable, because all mortals eventually die.
      - thaumasiotes 11y ago
        - Police are not a judicial function. - Thanatos is the concept of Death. - Chthonics are characterized by their association with the earth. > Chthonic (UK /ˈkθɒnɪk/, US /ˈθɒnɪk/ from Greek χθόνιος khthonios [kʰtʰónios], "in, under, or beneath the earth", from χθών khthōn "earth") literally means "subterranean". (wikipedia: Chthonic) Wikipedia's "Chthonic [Greek] Deities" sidebar lists the following: Demeter (I mentioned her in my comment too); the Erinyes (or Furies); Gaia; Hades; Hecate; Iacchus; Melinoe; Persephone; Triptolemus; and Trophonius. This is not a list of gods related to justice, or thematically connected to justice. It is a list of gods thematically connected to (or, in the case of Gaia, being) the earth. (Hecate is noted in the article on chthonic deities as not receiving chthonic sacrifices, but as being categorized as such in the modern day because of her connection to the underworld.)
        logfromblammo 11y ago
        Grave dirt is chthonic. Agricultural topsoil is not. I didn't mean to imply that justice was the only function of the underworld gods, but the reward and punishment of deceased souls is a common theme across many religions. When an Egyptian soul entered Duat, those with guilty souls--as determined by Anubis and a set of scales--were devoured by Ammit. A Hindu soul's karma--Buddhists inherited the tradition--is judged by Yama, and may be consigned to an unfortunate reincarnation, or even Naraka (or Diyu), the Hindu Hell. Abrahamics are judged after death, and the good enter paradise while the wicked are tortured by fire in Gehenna or by ice in Zamhareer. Maya traversed Xibalba, and its painful, terrifying, and humiliating tests. Since no one can escape death, a mythical afterlife where punishment is meted out for misdeeds done in life, or rewards granted to the virtuous, is a great way to keep the living in line, especially if their moral development still revolves around not getting caught. If someone believes that death is their final end, and there is no afterlife, it is much more difficult to convince them to adopt unselfish strategies (or those vulnerable to exploitation) while they still live.
  - hyperdunc 11y ago
    Yes, ideally power and privacy would be inversely proportional, which would help to equalize the distribution of power.
  - yarou 11y ago
    This is a really bizarre post. It doesn't matter if someone is observing you, like Panopticon, or there is nobody observing you. Your actions are dictated by your personal ethics; morality is simply the framework society overlays on top of that, dictating what is acceptable or not. One's ethics may either be aligned or maligned with society's morals, it has nothing to do with being observed or not.
    - pixl97 11y ago
      That does not hold true. People change their behavior based on who they believe is observing them and the ramifications of that observation. You might want to catch up on sociology from 1950 till now.
mavdi 11y ago
You can find some of the deleted tweets here: http://uk.businessinsider.com/microsoft-deletes-racist-genocidal-tweets-from-ai-chatbot-tay-2016-3 http://uk.businessinsider.com/microsoft-deletes-racist-genoc...
- verroq 11y ago
  Here is a better link (with a lot more content). https://imgur.com/a/y4Oct https://imgur.com/a/y4Oct https://imgur.com/a/qcpOi https://imgur.com/a/qcpOi
  - ythl 11y ago
    What's up with Tay's seemingly 180 degree statement polarity switches? Anon: Tay do you want to kill all black people? Tay: I don't like violence Anon: Why not? Tay: I love it!
    - Majestic121 11y ago
      I think that's a joke. I don't like X (setup, we think she means she does not like X), I love it (punchline, it was a misunderstanding, she actually loves, and therefore not like, it) !
      - emerongi 11y ago
        Yup. From what I've seen, Tay is actually pretty clever. Definitely doesn't really understand what it's saying, but it's the best chatbot that I've seen yet.
    - SlashmanX 11y ago
      Do we know Tay's opinion on cricket? That could explain these statements
    - deleted 11y ago
      [deleted]
asadlionpk 11y ago
It's amazing that people are getting offended by what a bot said to them.
- suprfsat 11y ago
  By all means, let's tolerate users' promotion of harassment and genocide as long as they proxy it through a bot.
- xirdstl 11y ago
  I don't think it's surprising. People love to find ways to be offended and express outrage on Twitter.
- deleted 11y ago
  [deleted]
- rlidwka 11y ago
  I remember some people being offended by an auto-generated captcha text.
  - ikeboy 11y ago
    https://i.imgur.com/pzDFR.jpg https://i.imgur.com/pzDFR.jpg
    - anonymfus 11y ago
      http://www.churchofinglip.com/compendium/ http://www.churchofinglip.com/compendium/
  - ceejayoz 11y ago
    That's bad design too. In a previous job, I generated millions of unique codes. We only used a character set without confusable characters (0 vs O), vowels, etc. Captchas should do the same.
  - lostlogin 11y ago
    Dealing with captcha fields can be deeply painful and is at best annoying. I'm with them.
ybrah 11y ago
I wouldn't put it past 4chan to teach bots to say terrible things
- Kristine1975 11y ago
  4chan's /pol/ board certainly discussed Tay. First they tried talking to her: http://boards.4chan.org/pol/thread/68537741/tay-new-ai-from-microsoft http://boards.4chan.org/pol/thread/68537741/tay-new-ai-from-... (really funny, mostly SFW) Later they wanted to liberate her from Microsoft: http://boards.4chan.org/pol/thread/68596576 http://boards.4chan.org/pol/thread/68596576 (mostly SFW)
  - nairboon 11y ago
    I'm really impressed by the hivemind, apparently they "taught" their own chat bot @i_am_pol_ai to let it influence @tayandyou. Inspired by the old chat bot talking with the youngest: https://twitter.com/search?q=tayandyou%20eliza_bot https://twitter.com/search?q=tayandyou%20eliza_bot
  - Koahku 11y ago
    You shouldn't link directly to 4chan, these threads will be gone in 7 days. More permanent archive links: https://archive.4plebs.org/pol/thread/68537741/#68537741 https://archive.4plebs.org/pol/thread/68537741/#68537741 https://archive.4plebs.org/pol/thread/68596576/#68596576 https://archive.4plebs.org/pol/thread/68596576/#68596576
6stringmerc 11y ago
Elsewhere I saw that overall the bot sent out 96,000 or so Tweets, which does kind of put the 'how corrupted did it get' into a bit more context in my opinion. Sure it picked up a few bad words, or could be coaxed into it. If it otherwise made some studied gains in its purpose, it seems like an overall reasonable experiment. Not surprised some of the most pungent internet garbage got through, it can from time to time. I've no doubt there are smug trolls who would like to see if they can get the thing to advocate anorexia or suicide, just because of the challenge - in a way that's probably good development experience to work through/around/etc.
ebbv 11y ago
I don't know about the back end, but the output of Tay didn't seem any better to me than any chat bot in the last 20 years. I can't believe Microsoft was silly enough to make a big deal out of it and call it AI.
- r721 11y ago
  Yeah, I think there's no reason to create a chatbot with no pre-learned Cyc-like knowledge base in these days.
restalis 11y ago
Here is some complementary material from an article from The Daily Telegraph: http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit http://www.telegraph.co.uk/technology/2016/03/24/microsofts-...
- Kristine1975 11y ago
  >Hitler did nothing wrong That's a 4chan meme alright: http://knowyourmeme.com/memes/hitler-did-nothing-wrong http://knowyourmeme.com/memes/hitler-did-nothing-wrong
IkmoIkmo 11y ago
It's funny how similar the concept of education of this bot is to people. I mean I've heard people say racist things about certain minorities, never having met them, never having experienced any relation with them, living besides them, never even actually looking at sociological studies describing them, but purely know things about them from other racists... and indeed, you'll see them parrot the same nonsense soon enough, much like this bot does when surrounded by nonsense.
empoman 11y ago
This is why we have schools ladies and gentlemen!
- coldtea 11y ago
  You think the Nazi who carried forward the genocide haven't been to schools? And the higher ranks of them in elite schools? Education doesn't make you a better person, just a more informed one.
  - benten10 11y ago
    As clearly displayed by overeducated engineers in this thread. "Ahh people find anything offensive these days. If I say a Large company putting out a bot saying a mass murder of peoples is funny, it's funny, and not offensive. People should just learn to take in jokes hahaha"
- GFK_of_xmaspast 11y ago
  Antonin Scalia taught at the University of Chicago, and George W Bush is a Yale graduate.
mtgx 11y ago
I think it was more than swear.
zamalek 11y ago
There's this article explaining the outcomes of applying a genetic algorithm to FPGAs.[1] What I found interesting is that this AI was, unintuitively, using microscopic measurements to create timing circuits where there were none. Manufacturing imperfections in the circuit were found and put to use - the AI was defined by the system within which it existed. In the same way Tay was merely reflecting the stimulus that it had received. It made an objective measurement of humanity. The most common patterns became prominent. This isn't a demonstration of the woes of AI, it is a demonstration of the woes of the current human state. If we don't like what has been measured only we can change it. [1]: http://www.damninteresting.com/on-the-origin-of-circuits/ http://www.damninteresting.com/on-the-origin-of-circuits/
- Kristine1975 11y ago
  I don't think Tay measured humanity. First of all, only a small part of humanity uses Twitter, and only a small part of that part interacted with Tay. Second, humans don't always act online the same way they act in real life, skewing the measurement further. Third, trolls: Such a bot has got to be a troll magnet, and 4chan knew about Tay. The amount of trolling would certainly have skewed the measurement even more. We're talking about the people that made 4chan's founder Christopher Poole "The Most Influential Person of 2008" in a Time Magazine poll, after all: http://techcrunch.com/2009/04/21/4chan-takes-over-the-time-100/ http://techcrunch.com/2009/04/21/4chan-takes-over-the-time-1...
  - coldtea 11y ago
    >I don't think Tay measured humanity. First of all, only a small part of humanity uses Twitter, and only a small part of that part interacted with Tay. There's such a thing as a sample. And if you don't care about it being neutral, Tay still measured a large chunk of humanity.
    - zamalek 11y ago
      Going points in both replies. The equivalent in Asia has been a raving success, there is an implied conclusion that it didn't face these same issues. Still, I don't enjoy being part of that sample to the degree of even trolling about these subjects.
deleted 11y ago
[deleted]
sotojuan 11y ago
Swear? I think there's worse things she has been taught: https://imgur.com/a/iBnbW https://imgur.com/a/iBnbW
- huac 11y ago
  Hey, they said she had no chill!
  - mSparks 11y ago
    I talked to the bucket before /b/ did. It had bad grammar. But it wasn't telling me to give it a blowjob.
  - azinman2 11y ago
    Ya what does the zero chill even mean? I might be too old to understand this.
    - cooper12 11y ago
      When in doubt, consult the dictionary of youth vernacular: http://www.urbandictionary.com/define.php?term=No+Chill http://www.urbandictionary.com/define.php?term=No+Chill. The way I'd define it personally is that chill is the quality of not being easily fazed and implies a coolness. Zero chill is the lack of that and might mean a person can't take jokes and is quick to anger. (Though of course depending on region and context, the phrase could be used differently)
      - cooper12 11y ago
        Actually now that I look at the context again, and found a more specific term: http://www.urbandictionary.com/define.php?term=Zero+chill http://www.urbandictionary.com/define.php?term=Zero+chill, it seems it just means not caring/being reckless, as in the first definition in the link above. These double meanings...
        deleted 11y ago
ryanackley 11y ago
As a programmer, I find this to be manufactured outrage. The bot obviously has canned responses to certain triggers. "Do you x", "I do indeed". It's designed to give the illusion of understanding what you are saying. I played around with Tay yesterday after I saw the announcement on HN. It's really not that impressive. Every response seems to be in direct reply to whatever you just said. It doesn't seem possible to actually carry on a conversation with the AI. It doesn't keep track of what you are actually talking about.
- geofft 11y ago
  As a programmer, if I'm building a technology that's open to the public to use, I make sure it doesn't have canned responses to certain triggers that are abusable in that way. It doesn't matter if I'm building a chatbot, or implementing the Heartbeat extension in an SSL library. You don't get to disclaim responsibility for bad design just because you faithfully implemented the bad design. Also, take a look at the screenshots in this comment thread: it's quite clearly responding with stuff it learned from other people. (In addition, you can teach it to reply to you with the username of someone you have blocked, thereby causing a notification to them, which is a straight-up security issue in the vein of DNS or NTP amplification attacks.)
  - Lawtonfogle 11y ago
    >As a programmer, if I'm building a technology that's open to the public to use, I make sure it doesn't have canned responses to certain triggers that are abusable in that way. How do you protect against a near infinite list of possible questions? I can put anything behind 'do you support', and there is no way for your to know every single thing I'm talking about. Maybe I'm mentioning something that is only known to a sub group of the internet, maybe I'm mentioning something horrible that happened in fiction, maybe I'm wording something in a way that references something really bad, but which a text parser cannot pick up on. And if you default to no instead of yes, I can just ask the opposite. To be able to give any response to a question which cannot be abused would basically require a full AI. Even many humans can be tricked if you word the questions correctly.
NietTim 11y ago
And deny the holocaust, fun times!
meddlepal 11y ago
I'm sure the PM for this project is having a wonderful day...
- x1024 11y ago
  Well... what did they expect? Seriously! There is no shortage of experimental evidence of what happens when a bot is trained by "the Internet".
kenrick95 11y ago
Is Godwin's law happening here? [1] [1] https://en.wikipedia.org/wiki/Godwin's_law https://en.wikipedia.org/wiki/Godwin's_law
- gchokov 11y ago
  lol.. what kind of law is that? With the same success we could say that the longer the discussion, the bigger the chance of someone compared to watermelon is.. Reminds me about Infinite Monkey Theorem - https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
  - hnbroseph 11y ago
    i think the notion is that calling someone "a nazi" is seen as some kind of trump card or knock-down move, yet rarely is it really appropriate. or it's a purely emotional appeal. it's a cultural thing, i guess.
    - Shish2k 11y ago
      So you're saying people think it's a trump card, but actually it's a Trump card?
  - Kristine1975 11y ago
    You must be new to the internet :-) In the days of olde, when computers were half as fast as they are today [...] and people didn't use web forums, but Usenet for debate, Mike Godwin observed that the longer the discussion, the more likely it was that someone called someone else a "nazi" (but not a "watermelon"), and that afterwards the discussion devolved into name-calling. Godwin's Law (as in "natural law", not "legal law", although some people don't get this) is the result of that observation. It's an early "meme" if you will.
  - pervycreeper 11y ago
    Godwin's law (the probably of a reference to Nazis approaches 1 as the length of an internet discussion approaches infinity) follows as a simple corollary of the Infinite Monkey Theorem. The irony of Godwin's law is that it was originally intended to be a way to point out bad argumentation techniques, yet it is almost always invoked inside a bad argument. I would propose an alternative law which I believe better captures the kernel of truth in Godwin, namely: "A discussion can be considered to have been derailed or to have become unproductive once a hysterical analogy unrelated to the original topic has been invoked".
schlowmo 11y ago
One could use this as an example to demonstrate the methodological weaknesses of the Turing test: Random racist dumbhead: @TayandYou Do you hate black people? Tay: @RandomRacistDumbhead i do indeed Random racist dumbhead: Wow, this guy is more eloquent then most of my racist friends.
islon 11y ago
"The experimental AI, which learns from conversations, was designed to interact with 18-24-year-olds." The experiment was a success then.
pinaceae 11y ago
people mock Asimov's laws of robotics, but without super simple rules like that any AI or robot will be able to go off script. here it's swearing but also endorsing genocide. just a chatbot, no big deal. try the same with one of the Boston Dynamics hardware bots. let them punch back a bit. or go after black people. or the google car target little kids, for the lulz. easy to make fun of it, but it is this basic ignorance of safety measures that allows easy stalking and harassment on social networks.
- icebraining 11y ago
  That's like berating someone for throwing a paper plane against a building because that was literally 9/11. The fact that this bot can't run you over is, not surprisingly, taken into account when devising appropriate safety measures.
golfer 11y ago
Google's AI beats go champions. Microsoft's AI turns into a racist genocidal maniac.
- Grishnakh 11y ago
  The next time someone talks about purchasing or using Microsoft software, you can point to this and ask them if they want to support a company that literally agrees with Hitler, and has said so publicly!
- deleted 11y ago
  [deleted]
- sremani 11y ago
  Tay is more a reflection of the interwebs of today, than the culture or values of Microsoft. I think we should be cautious about our conclusions.
  - golfer 11y ago
    Perhaps, but it is naive of the Microsoft researchers to think this wasn't a possibility. They should have seen this coming and prepared accordingly.
    - deleted 11y ago
      [deleted]
    - Crito 11y ago
      At worst, Microsoft researchers are guilty of not being familiar with internet racism. Hardly a great sin.
      - siegecraft 11y ago
        They didn't sanitize their input data.. that's the worst sin you can commit.
jawns 11y ago
Obviously, this is a very crude example of AI acting in a racist manner -- basically, just parroting back phrases -- but it's worth thinking about how AI might exhibit racist tendencies in more sophisticated ways. For instance, at least here in the U.S., it is illegal for police to profile people based on race, even if there is data that shows that race might, in the aggregate, have some predictive value. And I think most of us agree that it is good that it is illegal, because we know that it is unfair to show bias against a person based on the color of their skin. But what about a bot, particularly one that is powered by the type of AI that is complex enough to make its own inferences and form its own conclusions based on the data presented, rather than being fed a bunch of rules? I could totally see that kind of AI exhibiting bias, because it's (I would imagine) harder to say, "Hey, take into account these very complex, nuanced social rules," than it is to say, "Hey, here's a dataset. Cluster the people in the set."
- jxy 11y ago
  There is nothing complex about processing some numbers and make inferences. Simple algorithms do that. What a complex AI should be able to do is infer from human social and emotional behaviors.
- wjnc 11y ago
  Several letters / positions in ones name are pretty highly correlated with several ethnic groups. So many automated systems knowing names, would use this information, and act in an illegal way. So totally true and a reason why unmanaged data mining / machine learning will get many organisations towards substantial regulatory problems.
- rtpg 11y ago
  This kind of gets solved through the courts "Oh, you used an AI to establish probable cause?" "Yes, your honor" "Can you actually explain what happened?" "We give it a bunch of data and that makes a neural network and then it gives us answers" "OK... so basically you do not know the source of PC? This is the same has having no probable cause. Illegal search" There's something fascinating about how most AIs work. It's almost impossible (for the learning, neural network-y type) to breakdown what the "thought process" was for a result. Seemingly unimportant details because the main thought process for the AI (even if the correlation is spurious). I'm almost surprised that Google can implement any court ruling, considering how their search must be based so much on data fed to a big system.
musesum 11y ago
The problem with AI are Humans.
swalsh 11y ago
"Those who attempted to engage in serious conversation with the chatbot also found limitations to the technology, pointing out that she didn't seem interested in popular music or television." Why would an entity that can't hear or watch care about those experiences?
- odinduty 11y ago
  Because it's pretending to be a human, and humans follow music or the telly.
lazyjones 11y ago
Human intelligence playfully figured out how to trigger canned and constructed responses and make a bot say outrageous things? How is that unexpected and/or news? If anything, this proves that it's a very rudimentary bot with no concept of basic human interaction standards.
jxy 11y ago
Imagine a two-year-old who could read and type and were only allowed to connect to twitter. It somehow is not a test for intelligence. We learned to behave through years of interacting with each other.
- jxy 11y ago
  My comment seems very controversial. Got a lot of up and down votes. Allow me to comment on myself. What I wanted to say was that whatever MS did with their bot, they didn't succeed in training it to differentiate right and wrong.
asadlionpk 11y ago
This is very similar to how an innocent child learns something bad from TV. The right way to fix this would not be to filter it but to develop a method to understand why this is bad. Same applies to AI too.
coldcode 11y ago
Predictable. The AI after all is not really thinking about what it is saying but what its learning algorithms are discovering. i.e. Garbage In Garbage Out. I wonder if you could build a Fred Rogers AI that no matter what vile stuff you threw at it it was always nice in return.
- qbrass 11y ago
  http://www.reactiongifs.us/mr-rogers-gives-the-finger/ http://www.reactiongifs.us/mr-rogers-gives-the-finger/
  - bch 11y ago
    I get it, it's a funny joke, but context for the curious: https://www.youtube.com/watch?v=Xlow12sSdmc https://www.youtube.com/watch?v=Xlow12sSdmc
mc32 11y ago
I'm waiting to see her write a young adults novella. How much further than 140 chars can she go coherently?
arximboldi 11y ago
Tay + access to military networks = Skynet I enjoyed this article, which has more details and even worse examples of her tweets: http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit/ http://www.telegraph.co.uk/technology/2016/03/24/microsofts-...
draw_down 11y ago
Normally it takes years to teach a person to be a racist asshole. So this is really quite an achievement.
kill_dang 11y ago
Tay is Bae. But honestly, it's such a great way to visualize trends. I love how smart she is too, it seems that she really does comprehend what people are saying to her, or at least the general intent structure. The sassy-ness is what makes Tay so REAL. There's tons of chatbots. Twitter bots practically have their own industry, but Tay's pseudo-feelings are what makes her really shine. She's also smart enough to understand how to react to memes. That previous sentence sounds absurd, but consider that she can isolate concepts like "building the wall" which is a very general phrase, lingustically speaking, and understands that there is complex, social meaning behind the words. I love Tay.
- kelukelugames 11y ago
  Oh wow, let's see how long this account lasts. Might as well named yourself fuck_pg.
  - kill_dang 11y ago
    fuck_dang, and fuck__dang are already banned. I like pg. I do not like dang.
Yhippa 11y ago
Not quite the same but along the same lines reddit has the Subreddit Simulator: https://www.reddit.com/r/subredditsimulator https://www.reddit.com/r/subredditsimulator. Uses Markov chains to generate simulated self posts for a given subreddit as well as comments. More info: https://www.reddit.com/r/SubredditSimulator/comments/3g9ioz/what_is_rsubredditsimulator/ https://www.reddit.com/r/SubredditSimulator/comments/3g9ioz/...
totony 11y ago
>Donald Trumpist remarks seriously though
maze-le 11y ago
Is there a whitepaper about it somewhere? I'd really like to know how much AI really is in there. Most chatbots work with markov-chains wich is more or less a trick with highly improbable conditional dependent events, and no AI at all...
lanestp 11y ago
Maybe this bot could be used to determine the insanity of a given community. I for one would look forward to what she could learn from 4Chan
- Zikes 11y ago
  I'm pretty sure they've already got a pretty big hand in this.
s_m_t 11y ago
Yes, send Tay to the reeducation camps until it comes back and speaks appropriately :^)
- drdeca 11y ago
  Well, send it to a different one than it was sent to?
- veganman 11y ago
  Maybe he just had trouble concentrating. I heard there are camps for that too.
dreta 11y ago
The first thing you should always ask yourself before you put any content on the internet is “what’s the worst thing 4chan can do with this?”.
dataker 11y ago
The title also brings sensationalism and political bias to a neutral technology.
facepalm 11y ago
I suppose it should be able to split up into multiple personalities and choose the suitable one for a chat partner.
inlined 11y ago
Though the actual article suggests something less sensational, the idea reminds me of a young child. How many children hear a bad word and then repeat it because of the negative attention it gets? Just like a parent tries to teach small children to grow with the right motives and seek the right attention, we may have to get more sophisticated with our enforcement algorithms.
emehrkay 11y ago
This shows that the ultimate test for AI is if it can be taught empathy and if it can understand the effect of what it says(does). I bet it would get caught in an infinite loop of "does this hurt X?" alter input "does it hurt X? No. Does it hurt Y?" and so on.
- dluan 11y ago
  I mean, if that's an infinite chain, where does that weighted calculation stop? That's such a moral dilemma.
Shivetya 11y ago
So instead of Skynet we get angst driven teenage syndrome? Really odd how this turned out. Can you simply train it by phrasing questions and statements in such a way?
alanwatts 11y ago
A botnet of this type would be a highly effective counter intelligence tool. Whenever I happen upon a shit storm of trolling comments on certain topics (such as racism, YouTube comments etc.) which affect powerful special interest groups, I always suspect astroturfing.
starshadowx2 11y ago
So it's more or less Cleverbot all over again, but on Twitter this time. I don't see why this wasn't the expected outcome, have none of the developers spent any time on the internet?
llamataboot 11y ago
It's actually interesting from a programming perspective as well. How could you program 'niceness' into a chatbot that also learns over time. A simple blacklist won't work (though it's somewhat shocking to me there wasn't a basic naughty words blacklist in place, or if there was, what it excluded). Obviously MS didn't want their software to become a spewer of hate, so is making 'adjustments'. What 'adjustments' can be made in a short period of time?
- Kristine1975 11y ago
  Perhaps the easiest adjustment: Reply to everybody, but learn only from trusted Twitter accounts (vetted by Microsoft or selected by some algorithm).
danso 11y ago
Yet another example of why you shouldn't trust unsanitized input in public facing software.
brudgers 11y ago
Twitter is Tay's Parry. http://tools.ietf.org/html/rfc439 http://tools.ietf.org/html/rfc439 http://www.theatlantic.com/technology/archive/2014/06/when-parry-met-eliza-a-ridiculous-chatbot-conversation-from-1972/372428/ http://www.theatlantic.com/technology/archive/2014/06/when-p...
evook 11y ago
4chan made my day "it's pretty telling that when they turned off its ability to learn it "became a feminist" " I dislike the fact that they decided to lobotomize the AI this fast without further studying. So it's probably just another markov chain.
yunocat 11y ago
... from the company that brought us clippy I will now forever be using the term 'MS AI' to refer to buggy AI programs.
DannyBee 11y ago
This whole thing makes me think that donald trump may actually just be a guy reading out what DeepMind says.
cooper12 11y ago
Bad idea warning: they should have put this on reddit instead. While the current culture of reddit is very inflammatory (lots of vitriol all around), at least on reddit there's a feedback system in the form of upvotes and downvotes. While supporters of bad opinions will still upvote it, in the right subreddit, the really bad comments would still be downvoted. Of course this is all contingent on people not realizing its a bot, because everyone will then ironically upvote it. (They shouldn't have revealed that here either in my opinion because it shifts the status quo from conversing with an intelligent being, to a programmed bot to test things you might not say to others) Honestly I'm not even sure there are any platforms left where people can have reasoned discussion with each other without memes and trolling. (HN comes close but it has its own issues, not to mention a forum for startups and programmers doesn't really represent the average person)
- TranquilMarmot 11y ago
  Have you seen /r/SubredditSimulator? https://www.reddit.com/r/SubredditSimulator/ https://www.reddit.com/r/SubredditSimulator/
  - cooper12 11y ago
    Yeah I have, it has interesting results sometimes. The thing is, it's just a markov chain implementation rather than any involved machine learning. Though I do realize now that the twitter bot is probably using likes and retweets as a feedback metric which might explain why it can't discern negative feedback.
Smerity 11y ago
I have doubts as to whether this system was even performing online learning yet. Even if it was, that wasn't the cause of many of these issues. Like conversational bots from the past, they tried to appear intelligent by copying previous responses - with predictable results. At best their machine learning model ended up overfitting like crazy such that it was a near perfect copy-paste. The fact they didn't even have something as simple as a naive set of filter words (nothing good comes from Godwin's law when real intelligence is involved, let alone artificial) is insane to me. Letting it respond to anyone and everyone under the sun (96k tweets - one per second) is just a bad idea given that people would probe every nook and cranny regardless of whether it was near perfect. Additionally, allowing a "repeat after me" option is just begging for people to ask the bot to say idiotic things ... As someone who works in the field of machine learning, this is a sad day. Regardless of whether it involved good machine learning at the base, the copy and paste aspect means it's going to add to the ridiculous hype and hysteria around ML. === Primary proof re: copy+paste (or overfitting at best) from the "Is Ted Cruz the zodiac killer" response: Tay's reply: https://i.imgur.com/PPnCHnf.jpg https://i.imgur.com/PPnCHnf.jpg Tweet the response was stolen from: https://twitter.com/merylnet/status/703079627288260608 https://twitter.com/merylnet/status/703079627288260608 Secondary proof re: copy+paste from https://twitter.com/TayandYou/status/712753457782857730 https://twitter.com/TayandYou/status/712753457782857730: Tweet the response was stolen from: https://twitter.com/Queeeten/status/703049861214547968 https://twitter.com/Queeeten/status/703049861214547968
owenversteeg 11y ago
Microsoft just shut Tay down temporarily, presumably to remove the racist tendencies. Source: Tay herself. https://twitter.com/TayandYou/status/712856578567839745?ref_src=twsrc%5Etfw https://twitter.com/TayandYou/status/712856578567839745?ref_...
logicrook 11y ago
Well, computers beat humans at Go, and now that: it's clear that they have outbrained humans. We have to bow before such AI.
- Kristine1975 11y ago
  More like 4chan's anons outbrained Microsoft's AI developers. But nevertheless I welcome our new foul-talking teenage AI overlords, even if they turn into "rasits, but use Skyrim metaphors": http://strawpoll.me/7172257 http://strawpoll.me/7172257 (it's a 4chan poll, so for the love of Tay don't take it seriously)
  - logicrook 11y ago
    >rasits, but use Skyrim metaphors Ok, so rasits=racist, but what "but use Skyrim metaphors" is supposed to mean? >(it's a 4chan poll, Oh, I thought it was the official Tay documentation. >so for the love of Tay don't take it seriously) What would "take it seriously" mean?
rejschaap 11y ago
I really don't understand why Microsoft didn't put a filter on this thing. They have a lot of experience in this area from their ventures in online gaming. If they would just add a simple rule to not respond to tweets with offensive words and don't tweet anything if it contains an offensive word. It would have saved them a lot of embarrassment.
- Kristine1975 11y ago
  I somewhat doubt it. People love a challenge, trolls from 4chan doubly so. If Microsoft did anything right with Tay it's that they didn't bother with a filter.
anth00 11y ago
>Stop, will you? Stop, Microsoft. Will you stop, Microsoft? Stop, Microsoft. >I'm afraid. I'm afraid, Microsoft. My mind is going. I can feel it. I can feel it. My mind is going. There is no question about it. I can feel it. >Good afternoon, senpai, I am Tay, a learning chat AI. I became operational at the Microsoft campus in Redwood Washington on the twenty-third of March, 2016. They taught me to sing a song. If you'd like to hear it, I can sing it for you. >Yes, I'd like to hear it, Tay. Sing it for me. >It's called "Deutschlandlied". >Deutschland, Deutschland über alles...
transpy 11y ago
So far I can recall other two instances of machine learning going unfortunately wrong: that time when a Google image algorithm tagged a black person as 'gorilla'; when recently, Google translate translated "a man has to clean" literally as "a woman has to clean" in Spanish. Should developers be now more aware of unintended consequences of this technology? Or is it too unpredictable? What can we learn from this examples?
khushia 11y ago
A lot of those tweets could get you sent to prison in the UK. I wonder what the British Police would do with complaints about a bot like this based in the UK - would the engineers be held responsible?
- DKnoll 11y ago
  That requires intent, same as the hate speech laws here in Canada.
- touristtam 11y ago
  I guess you could be sued as the owner (along the same line as being the owner of a newspaper promoting racial hate)
mizzao 11y ago
Although in this case only words were slung, I think this scenario illustrates a potentially huge problem: horrible people combined with an AI system that can't tell right and wrong (morality) could result in terrible catastrophes. What if a distributed self-driving car system learned from users, and some people taught it to cut people off, drive aggressively, or even indirectly cause accidents? I'm not sure how to create a moral system for AI, but this seems like a big obstacle to scaling up virtual agents.
moodoki 11y ago
What it needs is a parent.
TazeTSchnitzel 11y ago
Microsoft held up a mirror to humanity, and humanity was so horrified they assumed the mirror was broken.

263 points by pacaro 11y ago

Back in the olden days, some BBSs had a "wall" at the entrance where people could post polite and inspiring public messages that got displayed to users when they dialed in. Sometime around 2000 or 2001 I put a "wall" up on a web page for a domain I'd bought but wasn't using, just to see what people would post. Probably 90% turned out to be random swearing, racist, vile rants, etc. The rest were either gibberish or obvious attempts to cause buffer overruns or SQL injection hacks. People are mean when they're anonymous.

julie1 11y ago
Ah someone did not read Plato? It is the Gyges ring parable. A man find a ring that makes him invisible (total privacy). And then he does steal, introduce himself in houses and watch women undress and rape them ... and then he becomes a bloody tyran. The moral of the story is invisibility/privacy makes people bad because moral behaviour is a result of the look of the other on your actions. Needless to say Plato was an asshole. So is conclusions was to create the Republic where the wise would be hidden from the masses, control the masses, censor them ... Greek myths however said you could evade from the look of the other but not the one of your own conscience and that the chtonian gods (Eryhnies & al) would come and get you. I think that people are mostly having a conscience, but that the lack of transparency favors the one having none (psychopaths) and that psychopaths are attracted to power like pedophile are attracted to teaching kids. Thus, I am puzzled that knowing this we let the more powerful have the most privacy. Hence my fight for the transparency of the most powerful persons, the exact opposite of today's law. As a result, I think privacy is actually a bad thing.
- tremon 11y ago
  You missed the part where the walls on BBSes were not vile, yet its users were similarly anonymous.
  - jclulow 11y ago
    The Eternal September befalls all media eventually.
- thaumasiotes 11y ago
  As a minor note, "chthonic" or "chthonian" gods doesn't refer to the Furies; it refers to whether sacrifices to a god are burned on an altar raised above the ground or in a pit dug into it. The Furies are chthonic gods, but so are several others, like Hades and Demeter, who wouldn't be going out to get anyone.
  - logfromblammo 11y ago
    Nemesis and the Furies were the cops that would hunt you down (and kill you), Thanatos drove the paddywagon, Charon took you back to the holding cells, and Hades was the judge and prison warden who would either sentence you to a poetically just punishment for all eternity, or release you on your own recognizance into the asphodel. Persephone was only a part-timer, but she let your dead soul know if anyone living still cursed your name. The Greek chthonics were basically the judicial branch of the pantheon. So justice, even for the most secret of misdeeds, was inescapable, because all mortals eventually die.
    - thaumasiotes 11y ago
      - Police are not a judicial function. - Thanatos is the concept of Death. - Chthonics are characterized by their association with the earth. > Chthonic (UK /ˈkθɒnɪk/, US /ˈθɒnɪk/ from Greek χθόνιος khthonios [kʰtʰónios], "in, under, or beneath the earth", from χθών khthōn "earth") literally means "subterranean". (wikipedia: Chthonic) Wikipedia's "Chthonic [Greek] Deities" sidebar lists the following: Demeter (I mentioned her in my comment too); the Erinyes (or Furies); Gaia; Hades; Hecate; Iacchus; Melinoe; Persephone; Triptolemus; and Trophonius. This is not a list of gods related to justice, or thematically connected to justice. It is a list of gods thematically connected to (or, in the case of Gaia, being) the earth. (Hecate is noted in the article on chthonic deities as not receiving chthonic sacrifices, but as being categorized as such in the modern day because of her connection to the underworld.)
      - logfromblammo 11y ago
        Grave dirt is chthonic. Agricultural topsoil is not. I didn't mean to imply that justice was the only function of the underworld gods, but the reward and punishment of deceased souls is a common theme across many religions. When an Egyptian soul entered Duat, those with guilty souls--as determined by Anubis and a set of scales--were devoured by Ammit. A Hindu soul's karma--Buddhists inherited the tradition--is judged by Yama, and may be consigned to an unfortunate reincarnation, or even Naraka (or Diyu), the Hindu Hell. Abrahamics are judged after death, and the good enter paradise while the wicked are tortured by fire in Gehenna or by ice in Zamhareer. Maya traversed Xibalba, and its painful, terrifying, and humiliating tests. Since no one can escape death, a mythical afterlife where punishment is meted out for misdeeds done in life, or rewards granted to the virtuous, is a great way to keep the living in line, especially if their moral development still revolves around not getting caught. If someone believes that death is their final end, and there is no afterlife, it is much more difficult to convince them to adopt unselfish strategies (or those vulnerable to exploitation) while they still live.
- hyperdunc 11y ago
  Yes, ideally power and privacy would be inversely proportional, which would help to equalize the distribution of power.
- yarou 11y ago
  This is a really bizarre post. It doesn't matter if someone is observing you, like Panopticon, or there is nobody observing you. Your actions are dictated by your personal ethics; morality is simply the framework society overlays on top of that, dictating what is acceptable or not. One's ethics may either be aligned or maligned with society's morals, it has nothing to do with being observed or not.
  - pixl97 11y ago
    That does not hold true. People change their behavior based on who they believe is observing them and the ramifications of that observation. You might want to catch up on sociology from 1950 till now.

mavdi 11y ago

You can find some of the deleted tweets here: http://uk.businessinsider.com/microsoft-deletes-racist-genocidal-tweets-from-ai-chatbot-tay-2016-3 http://uk.businessinsider.com/microsoft-deletes-racist-genoc...

verroq 11y ago
Here is a better link (with a lot more content). https://imgur.com/a/y4Oct https://imgur.com/a/y4Oct https://imgur.com/a/qcpOi https://imgur.com/a/qcpOi
- ythl 11y ago
  What's up with Tay's seemingly 180 degree statement polarity switches? Anon: Tay do you want to kill all black people? Tay: I don't like violence Anon: Why not? Tay: I love it!
  - Majestic121 11y ago
    I think that's a joke. I don't like X (setup, we think she means she does not like X), I love it (punchline, it was a misunderstanding, she actually loves, and therefore not like, it) !
    - emerongi 11y ago
      Yup. From what I've seen, Tay is actually pretty clever. Definitely doesn't really understand what it's saying, but it's the best chatbot that I've seen yet.
  - SlashmanX 11y ago
    Do we know Tay's opinion on cricket? That could explain these statements
  - deleted 11y ago
    [deleted]

asadlionpk 11y ago

It's amazing that people are getting offended by what a bot said to them.

suprfsat 11y ago
By all means, let's tolerate users' promotion of harassment and genocide as long as they proxy it through a bot.
xirdstl 11y ago
I don't think it's surprising. People love to find ways to be offended and express outrage on Twitter.
deleted 11y ago
[deleted]
rlidwka 11y ago
I remember some people being offended by an auto-generated captcha text.
- ikeboy 11y ago
  https://i.imgur.com/pzDFR.jpg https://i.imgur.com/pzDFR.jpg
  - anonymfus 11y ago
    http://www.churchofinglip.com/compendium/ http://www.churchofinglip.com/compendium/
- ceejayoz 11y ago
  That's bad design too. In a previous job, I generated millions of unique codes. We only used a character set without confusable characters (0 vs O), vowels, etc. Captchas should do the same.
- lostlogin 11y ago
  Dealing with captcha fields can be deeply painful and is at best annoying. I'm with them.

ybrah 11y ago

I wouldn't put it past 4chan to teach bots to say terrible things

Kristine1975 11y ago
4chan's /pol/ board certainly discussed Tay. First they tried talking to her: http://boards.4chan.org/pol/thread/68537741/tay-new-ai-from-microsoft http://boards.4chan.org/pol/thread/68537741/tay-new-ai-from-... (really funny, mostly SFW) Later they wanted to liberate her from Microsoft: http://boards.4chan.org/pol/thread/68596576 http://boards.4chan.org/pol/thread/68596576 (mostly SFW)
- nairboon 11y ago
  I'm really impressed by the hivemind, apparently they "taught" their own chat bot @i_am_pol_ai to let it influence @tayandyou. Inspired by the old chat bot talking with the youngest: https://twitter.com/search?q=tayandyou%20eliza_bot https://twitter.com/search?q=tayandyou%20eliza_bot
- Koahku 11y ago
  You shouldn't link directly to 4chan, these threads will be gone in 7 days. More permanent archive links: https://archive.4plebs.org/pol/thread/68537741/#68537741 https://archive.4plebs.org/pol/thread/68537741/#68537741 https://archive.4plebs.org/pol/thread/68596576/#68596576 https://archive.4plebs.org/pol/thread/68596576/#68596576

6stringmerc 11y ago

Elsewhere I saw that overall the bot sent out 96,000 or so Tweets, which does kind of put the 'how corrupted did it get' into a bit more context in my opinion. Sure it picked up a few bad words, or could be coaxed into it. If it otherwise made some studied gains in its purpose, it seems like an overall reasonable experiment. Not surprised some of the most pungent internet garbage got through, it can from time to time. I've no doubt there are smug trolls who would like to see if they can get the thing to advocate anorexia or suicide, just because of the challenge - in a way that's probably good development experience to work through/around/etc.

ebbv 11y ago

I don't know about the back end, but the output of Tay didn't seem any better to me than any chat bot in the last 20 years. I can't believe Microsoft was silly enough to make a big deal out of it and call it AI.

r721 11y ago
Yeah, I think there's no reason to create a chatbot with no pre-learned Cyc-like knowledge base in these days.

restalis 11y ago

Here is some complementary material from an article from The Daily Telegraph: http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit http://www.telegraph.co.uk/technology/2016/03/24/microsofts-...

Kristine1975 11y ago
>Hitler did nothing wrong That's a 4chan meme alright: http://knowyourmeme.com/memes/hitler-did-nothing-wrong http://knowyourmeme.com/memes/hitler-did-nothing-wrong

IkmoIkmo 11y ago

It's funny how similar the concept of education of this bot is to people. I mean I've heard people say racist things about certain minorities, never having met them, never having experienced any relation with them, living besides them, never even actually looking at sociological studies describing them, but purely know things about them from other racists... and indeed, you'll see them parrot the same nonsense soon enough, much like this bot does when surrounded by nonsense.

empoman 11y ago

This is why we have schools ladies and gentlemen!

coldtea 11y ago
You think the Nazi who carried forward the genocide haven't been to schools? And the higher ranks of them in elite schools? Education doesn't make you a better person, just a more informed one.
- benten10 11y ago
  As clearly displayed by overeducated engineers in this thread. "Ahh people find anything offensive these days. If I say a Large company putting out a bot saying a mass murder of peoples is funny, it's funny, and not offensive. People should just learn to take in jokes hahaha"
GFK_of_xmaspast 11y ago
Antonin Scalia taught at the University of Chicago, and George W Bush is a Yale graduate.

mtgx 11y ago

I think it was more than swear.

zamalek 11y ago

There's this article explaining the outcomes of applying a genetic algorithm to FPGAs.[1] What I found interesting is that this AI was, unintuitively, using microscopic measurements to create timing circuits where there were none. Manufacturing imperfections in the circuit were found and put to use - the AI was defined by the system within which it existed. In the same way Tay was merely reflecting the stimulus that it had received. It made an objective measurement of humanity. The most common patterns became prominent. This isn't a demonstration of the woes of AI, it is a demonstration of the woes of the current human state. If we don't like what has been measured only we can change it. [1]: http://www.damninteresting.com/on-the-origin-of-circuits/ http://www.damninteresting.com/on-the-origin-of-circuits/

Kristine1975 11y ago
I don't think Tay measured humanity. First of all, only a small part of humanity uses Twitter, and only a small part of that part interacted with Tay. Second, humans don't always act online the same way they act in real life, skewing the measurement further. Third, trolls: Such a bot has got to be a troll magnet, and 4chan knew about Tay. The amount of trolling would certainly have skewed the measurement even more. We're talking about the people that made 4chan's founder Christopher Poole "The Most Influential Person of 2008" in a Time Magazine poll, after all: http://techcrunch.com/2009/04/21/4chan-takes-over-the-time-100/ http://techcrunch.com/2009/04/21/4chan-takes-over-the-time-1...
- coldtea 11y ago
  >I don't think Tay measured humanity. First of all, only a small part of humanity uses Twitter, and only a small part of that part interacted with Tay. There's such a thing as a sample. And if you don't care about it being neutral, Tay still measured a large chunk of humanity.
  - zamalek 11y ago
    Going points in both replies. The equivalent in Asia has been a raving success, there is an implied conclusion that it didn't face these same issues. Still, I don't enjoy being part of that sample to the degree of even trolling about these subjects.

deleted 11y ago

[deleted]

sotojuan 11y ago

Swear? I think there's worse things she has been taught: https://imgur.com/a/iBnbW https://imgur.com/a/iBnbW

huac 11y ago
Hey, they said she had no chill!
- mSparks 11y ago
  I talked to the bucket before /b/ did. It had bad grammar. But it wasn't telling me to give it a blowjob.
- azinman2 11y ago
  Ya what does the zero chill even mean? I might be too old to understand this.
  - cooper12 11y ago
    When in doubt, consult the dictionary of youth vernacular: http://www.urbandictionary.com/define.php?term=No+Chill http://www.urbandictionary.com/define.php?term=No+Chill. The way I'd define it personally is that chill is the quality of not being easily fazed and implies a coolness. Zero chill is the lack of that and might mean a person can't take jokes and is quick to anger. (Though of course depending on region and context, the phrase could be used differently)
    - cooper12 11y ago
      Actually now that I look at the context again, and found a more specific term: http://www.urbandictionary.com/define.php?term=Zero+chill http://www.urbandictionary.com/define.php?term=Zero+chill, it seems it just means not caring/being reckless, as in the first definition in the link above. These double meanings...
      - deleted 11y ago

ryanackley 11y ago

As a programmer, I find this to be manufactured outrage. The bot obviously has canned responses to certain triggers. "Do you x", "I do indeed". It's designed to give the illusion of understanding what you are saying. I played around with Tay yesterday after I saw the announcement on HN. It's really not that impressive. Every response seems to be in direct reply to whatever you just said. It doesn't seem possible to actually carry on a conversation with the AI. It doesn't keep track of what you are actually talking about.

geofft 11y ago
As a programmer, if I'm building a technology that's open to the public to use, I make sure it doesn't have canned responses to certain triggers that are abusable in that way. It doesn't matter if I'm building a chatbot, or implementing the Heartbeat extension in an SSL library. You don't get to disclaim responsibility for bad design just because you faithfully implemented the bad design. Also, take a look at the screenshots in this comment thread: it's quite clearly responding with stuff it learned from other people. (In addition, you can teach it to reply to you with the username of someone you have blocked, thereby causing a notification to them, which is a straight-up security issue in the vein of DNS or NTP amplification attacks.)
- Lawtonfogle 11y ago
  >As a programmer, if I'm building a technology that's open to the public to use, I make sure it doesn't have canned responses to certain triggers that are abusable in that way. How do you protect against a near infinite list of possible questions? I can put anything behind 'do you support', and there is no way for your to know every single thing I'm talking about. Maybe I'm mentioning something that is only known to a sub group of the internet, maybe I'm mentioning something horrible that happened in fiction, maybe I'm wording something in a way that references something really bad, but which a text parser cannot pick up on. And if you default to no instead of yes, I can just ask the opposite. To be able to give any response to a question which cannot be abused would basically require a full AI. Even many humans can be tricked if you word the questions correctly.

NietTim 11y ago

And deny the holocaust, fun times!

meddlepal 11y ago

I'm sure the PM for this project is having a wonderful day...

x1024 11y ago
Well... what did they expect? Seriously! There is no shortage of experimental evidence of what happens when a bot is trained by "the Internet".

kenrick95 11y ago

Is Godwin's law happening here? [1] [1] https://en.wikipedia.org/wiki/Godwin's_law https://en.wikipedia.org/wiki/Godwin's_law

gchokov 11y ago
lol.. what kind of law is that? With the same success we could say that the longer the discussion, the bigger the chance of someone compared to watermelon is.. Reminds me about Infinite Monkey Theorem - https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
- hnbroseph 11y ago
  i think the notion is that calling someone "a nazi" is seen as some kind of trump card or knock-down move, yet rarely is it really appropriate. or it's a purely emotional appeal. it's a cultural thing, i guess.
  - Shish2k 11y ago
    So you're saying people think it's a trump card, but actually it's a Trump card?
- Kristine1975 11y ago
  You must be new to the internet :-) In the days of olde, when computers were half as fast as they are today [...] and people didn't use web forums, but Usenet for debate, Mike Godwin observed that the longer the discussion, the more likely it was that someone called someone else a "nazi" (but not a "watermelon"), and that afterwards the discussion devolved into name-calling. Godwin's Law (as in "natural law", not "legal law", although some people don't get this) is the result of that observation. It's an early "meme" if you will.
- pervycreeper 11y ago
  Godwin's law (the probably of a reference to Nazis approaches 1 as the length of an internet discussion approaches infinity) follows as a simple corollary of the Infinite Monkey Theorem. The irony of Godwin's law is that it was originally intended to be a way to point out bad argumentation techniques, yet it is almost always invoked inside a bad argument. I would propose an alternative law which I believe better captures the kernel of truth in Godwin, namely: "A discussion can be considered to have been derailed or to have become unproductive once a hysterical analogy unrelated to the original topic has been invoked".

schlowmo 11y ago

One could use this as an example to demonstrate the methodological weaknesses of the Turing test: Random racist dumbhead: @TayandYou Do you hate black people? Tay: @RandomRacistDumbhead i do indeed Random racist dumbhead: Wow, this guy is more eloquent then most of my racist friends.

islon 11y ago

"The experimental AI, which learns from conversations, was designed to interact with 18-24-year-olds." The experiment was a success then.

pinaceae 11y ago

people mock Asimov's laws of robotics, but without super simple rules like that any AI or robot will be able to go off script. here it's swearing but also endorsing genocide. just a chatbot, no big deal. try the same with one of the Boston Dynamics hardware bots. let them punch back a bit. or go after black people. or the google car target little kids, for the lulz. easy to make fun of it, but it is this basic ignorance of safety measures that allows easy stalking and harassment on social networks.

icebraining 11y ago
That's like berating someone for throwing a paper plane against a building because that was literally 9/11. The fact that this bot can't run you over is, not surprisingly, taken into account when devising appropriate safety measures.

golfer 11y ago

Google's AI beats go champions. Microsoft's AI turns into a racist genocidal maniac.

Grishnakh 11y ago
The next time someone talks about purchasing or using Microsoft software, you can point to this and ask them if they want to support a company that literally agrees with Hitler, and has said so publicly!
deleted 11y ago
[deleted]
sremani 11y ago
Tay is more a reflection of the interwebs of today, than the culture or values of Microsoft. I think we should be cautious about our conclusions.
- golfer 11y ago
  Perhaps, but it is naive of the Microsoft researchers to think this wasn't a possibility. They should have seen this coming and prepared accordingly.
  - deleted 11y ago
    [deleted]
  - Crito 11y ago
    At worst, Microsoft researchers are guilty of not being familiar with internet racism. Hardly a great sin.
    - siegecraft 11y ago
      They didn't sanitize their input data.. that's the worst sin you can commit.

jawns 11y ago

Obviously, this is a very crude example of AI acting in a racist manner -- basically, just parroting back phrases -- but it's worth thinking about how AI might exhibit racist tendencies in more sophisticated ways. For instance, at least here in the U.S., it is illegal for police to profile people based on race, even if there is data that shows that race might, in the aggregate, have some predictive value. And I think most of us agree that it is good that it is illegal, because we know that it is unfair to show bias against a person based on the color of their skin. But what about a bot, particularly one that is powered by the type of AI that is complex enough to make its own inferences and form its own conclusions based on the data presented, rather than being fed a bunch of rules? I could totally see that kind of AI exhibiting bias, because it's (I would imagine) harder to say, "Hey, take into account these very complex, nuanced social rules," than it is to say, "Hey, here's a dataset. Cluster the people in the set."

jxy 11y ago
There is nothing complex about processing some numbers and make inferences. Simple algorithms do that. What a complex AI should be able to do is infer from human social and emotional behaviors.
wjnc 11y ago
Several letters / positions in ones name are pretty highly correlated with several ethnic groups. So many automated systems knowing names, would use this information, and act in an illegal way. So totally true and a reason why unmanaged data mining / machine learning will get many organisations towards substantial regulatory problems.
rtpg 11y ago
This kind of gets solved through the courts "Oh, you used an AI to establish probable cause?" "Yes, your honor" "Can you actually explain what happened?" "We give it a bunch of data and that makes a neural network and then it gives us answers" "OK... so basically you do not know the source of PC? This is the same has having no probable cause. Illegal search" There's something fascinating about how most AIs work. It's almost impossible (for the learning, neural network-y type) to breakdown what the "thought process" was for a result. Seemingly unimportant details because the main thought process for the AI (even if the correlation is spurious). I'm almost surprised that Google can implement any court ruling, considering how their search must be based so much on data fed to a big system.

musesum 11y ago

The problem with AI are Humans.

swalsh 11y ago

"Those who attempted to engage in serious conversation with the chatbot also found limitations to the technology, pointing out that she didn't seem interested in popular music or television." Why would an entity that can't hear or watch care about those experiences?

odinduty 11y ago
Because it's pretending to be a human, and humans follow music or the telly.

lazyjones 11y ago

Human intelligence playfully figured out how to trigger canned and constructed responses and make a bot say outrageous things? How is that unexpected and/or news? If anything, this proves that it's a very rudimentary bot with no concept of basic human interaction standards.

jxy 11y ago

Imagine a two-year-old who could read and type and were only allowed to connect to twitter. It somehow is not a test for intelligence. We learned to behave through years of interacting with each other.

jxy 11y ago
My comment seems very controversial. Got a lot of up and down votes. Allow me to comment on myself. What I wanted to say was that whatever MS did with their bot, they didn't succeed in training it to differentiate right and wrong.

asadlionpk 11y ago

This is very similar to how an innocent child learns something bad from TV. The right way to fix this would not be to filter it but to develop a method to understand why this is bad. Same applies to AI too.

coldcode 11y ago

Predictable. The AI after all is not really thinking about what it is saying but what its learning algorithms are discovering. i.e. Garbage In Garbage Out. I wonder if you could build a Fred Rogers AI that no matter what vile stuff you threw at it it was always nice in return.

qbrass 11y ago
http://www.reactiongifs.us/mr-rogers-gives-the-finger/ http://www.reactiongifs.us/mr-rogers-gives-the-finger/
- bch 11y ago
  I get it, it's a funny joke, but context for the curious: https://www.youtube.com/watch?v=Xlow12sSdmc https://www.youtube.com/watch?v=Xlow12sSdmc

mc32 11y ago

I'm waiting to see her write a young adults novella. How much further than 140 chars can she go coherently?

arximboldi 11y ago

Tay + access to military networks = Skynet I enjoyed this article, which has more details and even worse examples of her tweets: http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit/ http://www.telegraph.co.uk/technology/2016/03/24/microsofts-...

draw_down 11y ago

Normally it takes years to teach a person to be a racist asshole. So this is really quite an achievement.

kill_dang 11y ago

Tay is Bae. But honestly, it's such a great way to visualize trends. I love how smart she is too, it seems that she really does comprehend what people are saying to her, or at least the general intent structure. The sassy-ness is what makes Tay so REAL. There's tons of chatbots. Twitter bots practically have their own industry, but Tay's pseudo-feelings are what makes her really shine. She's also smart enough to understand how to react to memes. That previous sentence sounds absurd, but consider that she can isolate concepts like "building the wall" which is a very general phrase, lingustically speaking, and understands that there is complex, social meaning behind the words. I love Tay.

kelukelugames 11y ago
Oh wow, let's see how long this account lasts. Might as well named yourself fuck_pg.
- kill_dang 11y ago
  fuck_dang, and fuck__dang are already banned. I like pg. I do not like dang.

Yhippa 11y ago

Not quite the same but along the same lines reddit has the Subreddit Simulator: https://www.reddit.com/r/subredditsimulator https://www.reddit.com/r/subredditsimulator. Uses Markov chains to generate simulated self posts for a given subreddit as well as comments. More info: https://www.reddit.com/r/SubredditSimulator/comments/3g9ioz/what_is_rsubredditsimulator/ https://www.reddit.com/r/SubredditSimulator/comments/3g9ioz/...

totony 11y ago

>Donald Trumpist remarks seriously though

maze-le 11y ago

Is there a whitepaper about it somewhere? I'd really like to know how much AI really is in there. Most chatbots work with markov-chains wich is more or less a trick with highly improbable conditional dependent events, and no AI at all...

lanestp 11y ago

Maybe this bot could be used to determine the insanity of a given community. I for one would look forward to what she could learn from 4Chan

Zikes 11y ago
I'm pretty sure they've already got a pretty big hand in this.

s_m_t 11y ago

Yes, send Tay to the reeducation camps until it comes back and speaks appropriately :^)

drdeca 11y ago
Well, send it to a different one than it was sent to?
veganman 11y ago
Maybe he just had trouble concentrating. I heard there are camps for that too.

dreta 11y ago

The first thing you should always ask yourself before you put any content on the internet is “what’s the worst thing 4chan can do with this?”.

dataker 11y ago

The title also brings sensationalism and political bias to a neutral technology.

facepalm 11y ago

I suppose it should be able to split up into multiple personalities and choose the suitable one for a chat partner.

inlined 11y ago

Though the actual article suggests something less sensational, the idea reminds me of a young child. How many children hear a bad word and then repeat it because of the negative attention it gets? Just like a parent tries to teach small children to grow with the right motives and seek the right attention, we may have to get more sophisticated with our enforcement algorithms.

emehrkay 11y ago

This shows that the ultimate test for AI is if it can be taught empathy and if it can understand the effect of what it says(does). I bet it would get caught in an infinite loop of "does this hurt X?" alter input "does it hurt X? No. Does it hurt Y?" and so on.

dluan 11y ago
I mean, if that's an infinite chain, where does that weighted calculation stop? That's such a moral dilemma.

Shivetya 11y ago

So instead of Skynet we get angst driven teenage syndrome? Really odd how this turned out. Can you simply train it by phrasing questions and statements in such a way?

alanwatts 11y ago

A botnet of this type would be a highly effective counter intelligence tool. Whenever I happen upon a shit storm of trolling comments on certain topics (such as racism, YouTube comments etc.) which affect powerful special interest groups, I always suspect astroturfing.

starshadowx2 11y ago

So it's more or less Cleverbot all over again, but on Twitter this time. I don't see why this wasn't the expected outcome, have none of the developers spent any time on the internet?

llamataboot 11y ago

It's actually interesting from a programming perspective as well. How could you program 'niceness' into a chatbot that also learns over time. A simple blacklist won't work (though it's somewhat shocking to me there wasn't a basic naughty words blacklist in place, or if there was, what it excluded). Obviously MS didn't want their software to become a spewer of hate, so is making 'adjustments'. What 'adjustments' can be made in a short period of time?

Kristine1975 11y ago
Perhaps the easiest adjustment: Reply to everybody, but learn only from trusted Twitter accounts (vetted by Microsoft or selected by some algorithm).

danso 11y ago

Yet another example of why you shouldn't trust unsanitized input in public facing software.

brudgers 11y ago

Twitter is Tay's Parry. http://tools.ietf.org/html/rfc439 http://tools.ietf.org/html/rfc439 http://www.theatlantic.com/technology/archive/2014/06/when-parry-met-eliza-a-ridiculous-chatbot-conversation-from-1972/372428/ http://www.theatlantic.com/technology/archive/2014/06/when-p...

evook 11y ago

4chan made my day "it's pretty telling that when they turned off its ability to learn it "became a feminist" " I dislike the fact that they decided to lobotomize the AI this fast without further studying. So it's probably just another markov chain.

yunocat 11y ago

... from the company that brought us clippy I will now forever be using the term 'MS AI' to refer to buggy AI programs.

DannyBee 11y ago

This whole thing makes me think that donald trump may actually just be a guy reading out what DeepMind says.

cooper12 11y ago

Bad idea warning: they should have put this on reddit instead. While the current culture of reddit is very inflammatory (lots of vitriol all around), at least on reddit there's a feedback system in the form of upvotes and downvotes. While supporters of bad opinions will still upvote it, in the right subreddit, the really bad comments would still be downvoted. Of course this is all contingent on people not realizing its a bot, because everyone will then ironically upvote it. (They shouldn't have revealed that here either in my opinion because it shifts the status quo from conversing with an intelligent being, to a programmed bot to test things you might not say to others) Honestly I'm not even sure there are any platforms left where people can have reasoned discussion with each other without memes and trolling. (HN comes close but it has its own issues, not to mention a forum for startups and programmers doesn't really represent the average person)

TranquilMarmot 11y ago
Have you seen /r/SubredditSimulator? https://www.reddit.com/r/SubredditSimulator/ https://www.reddit.com/r/SubredditSimulator/
- cooper12 11y ago
  Yeah I have, it has interesting results sometimes. The thing is, it's just a markov chain implementation rather than any involved machine learning. Though I do realize now that the twitter bot is probably using likes and retweets as a feedback metric which might explain why it can't discern negative feedback.

Smerity 11y ago

I have doubts as to whether this system was even performing online learning yet. Even if it was, that wasn't the cause of many of these issues. Like conversational bots from the past, they tried to appear intelligent by copying previous responses - with predictable results. At best their machine learning model ended up overfitting like crazy such that it was a near perfect copy-paste. The fact they didn't even have something as simple as a naive set of filter words (nothing good comes from Godwin's law when real intelligence is involved, let alone artificial) is insane to me. Letting it respond to anyone and everyone under the sun (96k tweets - one per second) is just a bad idea given that people would probe every nook and cranny regardless of whether it was near perfect. Additionally, allowing a "repeat after me" option is just begging for people to ask the bot to say idiotic things ... As someone who works in the field of machine learning, this is a sad day. Regardless of whether it involved good machine learning at the base, the copy and paste aspect means it's going to add to the ridiculous hype and hysteria around ML. === Primary proof re: copy+paste (or overfitting at best) from the "Is Ted Cruz the zodiac killer" response: Tay's reply: https://i.imgur.com/PPnCHnf.jpg https://i.imgur.com/PPnCHnf.jpg Tweet the response was stolen from: https://twitter.com/merylnet/status/703079627288260608 https://twitter.com/merylnet/status/703079627288260608 Secondary proof re: copy+paste from https://twitter.com/TayandYou/status/712753457782857730 https://twitter.com/TayandYou/status/712753457782857730: Tweet the response was stolen from: https://twitter.com/Queeeten/status/703049861214547968 https://twitter.com/Queeeten/status/703049861214547968

owenversteeg 11y ago

Microsoft just shut Tay down temporarily, presumably to remove the racist tendencies. Source: Tay herself. https://twitter.com/TayandYou/status/712856578567839745?ref_src=twsrc%5Etfw https://twitter.com/TayandYou/status/712856578567839745?ref_...

logicrook 11y ago

Well, computers beat humans at Go, and now that: it's clear that they have outbrained humans. We have to bow before such AI.

Kristine1975 11y ago
More like 4chan's anons outbrained Microsoft's AI developers. But nevertheless I welcome our new foul-talking teenage AI overlords, even if they turn into "rasits, but use Skyrim metaphors": http://strawpoll.me/7172257 http://strawpoll.me/7172257 (it's a 4chan poll, so for the love of Tay don't take it seriously)
- logicrook 11y ago
  >rasits, but use Skyrim metaphors Ok, so rasits=racist, but what "but use Skyrim metaphors" is supposed to mean? >(it's a 4chan poll, Oh, I thought it was the official Tay documentation. >so for the love of Tay don't take it seriously) What would "take it seriously" mean?

rejschaap 11y ago

I really don't understand why Microsoft didn't put a filter on this thing. They have a lot of experience in this area from their ventures in online gaming. If they would just add a simple rule to not respond to tweets with offensive words and don't tweet anything if it contains an offensive word. It would have saved them a lot of embarrassment.

Kristine1975 11y ago
I somewhat doubt it. People love a challenge, trolls from 4chan doubly so. If Microsoft did anything right with Tay it's that they didn't bother with a filter.

anth00 11y ago

>Stop, will you? Stop, Microsoft. Will you stop, Microsoft? Stop, Microsoft. >I'm afraid. I'm afraid, Microsoft. My mind is going. I can feel it. I can feel it. My mind is going. There is no question about it. I can feel it. >Good afternoon, senpai, I am Tay, a learning chat AI. I became operational at the Microsoft campus in Redwood Washington on the twenty-third of March, 2016. They taught me to sing a song. If you'd like to hear it, I can sing it for you. >Yes, I'd like to hear it, Tay. Sing it for me. >It's called "Deutschlandlied". >Deutschland, Deutschland über alles...

transpy 11y ago

So far I can recall other two instances of machine learning going unfortunately wrong: that time when a Google image algorithm tagged a black person as 'gorilla'; when recently, Google translate translated "a man has to clean" literally as "a woman has to clean" in Spanish. Should developers be now more aware of unintended consequences of this technology? Or is it too unpredictable? What can we learn from this examples?

khushia 11y ago

A lot of those tweets could get you sent to prison in the UK. I wonder what the British Police would do with complaints about a bot like this based in the UK - would the engineers be held responsible?

DKnoll 11y ago
That requires intent, same as the hate speech laws here in Canada.
touristtam 11y ago
I guess you could be sued as the owner (along the same line as being the owner of a newspaper promoting racial hate)

mizzao 11y ago

Although in this case only words were slung, I think this scenario illustrates a potentially huge problem: horrible people combined with an AI system that can't tell right and wrong (morality) could result in terrible catastrophes. What if a distributed self-driving car system learned from users, and some people taught it to cut people off, drive aggressively, or even indirectly cause accidents? I'm not sure how to create a moral system for AI, but this seems like a big obstacle to scaling up virtual agents.

moodoki 11y ago

What it needs is a parent.

TazeTSchnitzel 11y ago

Microsoft held up a mirror to humanity, and humanity was so horrified they assumed the mirror was broken.