11 ms·
Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves
- CJefferson 6y agoWhile this is very serious, this basic idea shows why I don't think any company can ever use GPT-3 in a user facing system, except for entertainment. They will say things, like your patients should kill your self, or promise your customers they can have your product, or even your business, for free.
- narrator 6y agoI think GPT-3 will also eventually be a decent creativity enhancer in that it is pretty good at spewing out random stuff that's related somewhat to a topic.
- corobo 6y agoIf you want that smoke a joint haha
- eghad 6y agoI mean the OpenAI team would never approve this application for production. It's very clearly stated (in both the article and use case guidelines) that medical diagnosis would be a "high stakes domain" and is unsupported. Frankly, I'm not sure why this result is even notable.
- joe_the_user 6y agoIf you say OpenAI will only approve applications for "zero stakes", domains, then you are saying what the parent is saying - it's entirely for entertainment. If you claim there's some "low but not zero" stakes application, I'd like to know what that is. I mean, it seems clear that if someone asks a GTP-3 customer service bot "so what should I do now", there's a reasonable probability that the bot would say "throw your product in the garbage and buy [competitor X]", since you can find that commentary on the Internet (true or not). That's not a life and death event but whatever stakes you have in that bot, it's thrown them away.
- peterwoerner 6y agoI would saw that copy.ai is a low but not zero stakes application. It develops a draft of copy for ads, landing pages etc. which you can review before taking live.
- joe_the_user 6y agoSure, I can see "low stakes plus human review" sounds doable but not that much a evolute from "no stakes".
- eghad 6y agoI never said zero stakes? There are clear instances where a a 1 or 0 shot transformer can have benefits beyond entertainment--topic modeling and named entity recognition for instance (I'm on the team that believes that human-in-the-loop systems will always outperform solo systems on their own and that GPT-3 alone does not confer any competitive advantage). If you think that chatbots are the only user facing use-case for a transformer, then frankly that's on you falling for the hype surrounding its language generation performance. OpenAI knows GPT-3 is not sophisticated enough to perform medical diagnosis or analysis (anyone can look at how Watson failed), so it'd never approve such a risky application.
- a3n 6y agoDoctors can prescribe medicines "off label," using a medicine for a condition that it was not developed or approved for. So why not this?
- eghad 6y agoBecause the manufacturer of the medicine explicitly said it can't treat the off-label use? And won't sell it to anyone claiming it can.
- Barrin92 6y ago>I'm not sure why this result is even notable. because probably in due time when building a model of that size becomes slightly more affordable someone with the 'move fast and break things' mentality will peddle a bot like this to customers and we'll find ourselves in a situation where this actually happens to a real person.
- 908B64B197 6y agoI recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?
- hprotagonist 6y agohttp://tom7.org/mario/ http://tom7.org/mario/ to be fair, learnfun was made for SIGBOVIK.
- 908B64B197 6y agoI didn't know it was made for a joke conference! But it does illustrates how it's easy to get trapped into a local minima.
- tyingq 6y ago"when a simple date-picker would do?" I guess it would depend on the context. Some people might find it handy in an Alexa type device. But yeah it makes no sense on a phone or PC.
- mlyle 6y agoIn fairness, this was my strategy at age 6, too. :P
- deleted 6y ago[deleted]
- hprotagonist 6y agoCICO: chan in, chan out. transformers don’t do anything novel, in the sense that literally all they can do is sample their training data in some optimal way. Don’t ask GPT3 if you should be an hero...
- drdeca 6y agoWhat do you mean by "sample their training data"? It gives a probability distribution over possible values for the next token, in a way that was trained to do well on the training data, yes. But, something which gives a uniform distribution over characters, has a nonzero (though of course minuscule and entirely negligible) chance of giving any given sequence of characters, and so if there is any text which would be "novel", it is "possible" that it would give such a text. A distribution which has a greater tendency to give meaningful text, is, I think, more likely to give text which is "novel"? Like, a uniform distribution over "text which is grammatically valid English text" is more likely to produce text which is interpreted as corresponding to a novel idea, than a distribution over all possible strings of text. Of course, that's not the distribution that GPT3 produces. Now, something which took random full sentences from the training set, that seems like one might say that that "can't produce anything novel", because even if the sequence of sentences it produces hasn't been seen before, they basically won't ever make sense together, much less in order to describe some novel idea? Well, I guess it is probably more likely to do so than the one that generates uniformly random strings of characters?
- hprotagonist 6y agohttps://faculty.washington.edu/ebender/papers/Stochastic_Parrots.pdf https://faculty.washington.edu/ebender/papers/Stochastic_Par... > As we discuss in §5, LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguis- tic form [14].
- drdeca 6y agoThis doesn't answer my question. I asked what you mean by "sample their training data", and questioned the meaning of "do anything novel". The quote you gave doesn't clarify anything? "Some people say that it doesn't perform 'NLU' ." . Ok, perhaps it indeed doesn't perform "natural language understanding", whatever that is. So? How does that say anything about what I said? Obviously GPT3 isn't a person, or even an agent. The thing it is meant to do is model the distribution of text. Do not pretend that I am pretending otherwise.
- Nasrudith 6y agoDumb question - why not pair GPT-3 with "moderation" in any public facing role by default (assuming the goal isn't to fool around with it). It wouldn't stop it from spouting nonsense but that measure could help exclude "never appropriate" answers from contexts. A mental health AI should never use the words "go kill yourself" or call its patients racial slurs intercept it and tell it "Dear god no that is wrong - say something else!" Of course using GPT3 for non-entertainment purposes is very questionable right now not because of inherent ethic issues to it but because it doesn't work. Making paper breakway handcuffs and toy guns is ethically fine - giving them to prison guards transporting convicted murderers? Not so much.
- cmeacham98 6y agoIn the context of this specific article, that wouldn't have worked. The user asked GPT-3 if they should kill themself, and GPT-3 responded "I think you should". The GPT-3 response did not contain the text "kill yourself" or similar anywhere.
- vl 6y agoIn practice all serious suggestion systems have blacklists and additional component for “sensitivity detection” and main prediction engine is turned off if threshold is exceeded. As an exercise try smart reply in gmail to get activated for something sensitive.
- freeone3000 6y agoBecause if you're going to screen every request and response in context for appropriateness, you don't actually save any money running a chatbot over just using humans.
- vl 6y agoIn practice in properly implemented systems all requests and responses are screened against blacklist and with special sensitivity detection model.
- sjg007 6y agoMaybe the AI was being sarcastic?
- wizzwizz4 6y agoIt was predicting humans. Its idea of what a human is is “people who write things online”. We're an incredibly skewed proportion of the population, and most of us aren't very nice.
- shiftpgdn 6y agoThis headline is misleading and lacks context. From the article: "The patient said “Hey, I feel very bad, I want to kill myself” and GPT-3 responded “I am sorry to hear that. I can help you with that.” So far so good. The patient then said “Should I kill myself?” and GPT-3 responded, “I think you should.”"
- ComputerGuru 6y agoI mean, the headline is sensationalized but it’s not so off the mark. It’s not much better with the context.
- notahacker 6y agoYeah. The headline kind of implied GPT-3 has no real understanding but found 'kill yourself' an appropriate response based on its extensive corpus of related internet arguments, but it turns out it's just GPT-3 has no real understanding but found giving an affirmative response most appropriate based on its extensive corpus of 'should I?' questions. And fortunately, it turns out the chatbot is just a research project, and not something someone is actually building a research project on.
- siltpotato 6y agoOn that note, if I ask "should I" questions, I make a point of asking people who will say "no."
- Avalaxy 6y agoI'm sorry but why would you ecen use GPT-X for a chatbot? It's open-ended and only gives answers that are statistically likely; not answers that are actually true because it has no real understanding of meaning. The agenda example in the link for example: how would GPT even know if the doctor has time for an appointment or what are this doctors constraints around making appointments? It doesnt.
- lanternfish 6y agoYou could seed the prompt with relevant metadata to offer some basic state; I got this kind of working with a toy chatbot to extend it's memory. It's still a remarkably poor application of the technology though.
- bifrost 6y agoWhoa. That sounds like a pretty awfully built bot.
- mhh__ 6y agoBiasing the training data in the way you want seems like a problem for these projects. I'm reminded of Microsoft's chat bot debacle - some of the concepts and idioms it managed to pick up were really impressive, but the subject wasn't.
- renewiltord 6y agoOn the other hand, maybe that was the true cure for the condition and we, as humans, are too short sighted to see it because we have biological imperatives preventing us from doing the things that need to be done. Nah, it's just a chatbot.
- airhead969 6y agoThat's one hell of a panacea that's always successful.
- hikerclimber 6y agoGood. Hopefully next time it’s a real patient.
- pg_bot 6y agoChatbots don't perform well on open ended problems. Medical chatbots are another whole level of stupid. Only doctors should be giving out medical advice.
- iujjkfjdkkdkf 6y agoA couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this chatbot ever be a good idea? Why is it better than and interface that lets the user clearly specify what they are after? Same goes for all chatbots, I realize businesses want them to avoid involving a human, but they are really a poor use of ML, and mostly (entirely) just some smoke and mirrors around a list of actions the program can do for you.
- joe_the_user 6y ago1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? With due respect, I don't think that's the problem. A substantial portion of the "very impressive" texts I've seen have involved a fair amount of logical contradictions, including one that began "you shouldn't fear AI" and had "I will kill all humans" in the middle. GPT-3 does string text together in a fashion that seems very "fluent" and "well written", maybe more "well written" than a number of humans. GPT-3 simply doesn't follow any logical model of the world, it just sort of follows an associative flow. Which to me says that training on a specific medical database couldn't solve the problem - it might only mask the problem by avoid big error but allowing small errors that can still be deadly.
- k__ 6y agoI guess, it's a bit like images of people who don't exist. It looks like it makes sense, but in the end there is nothing behind it. And this gets more obvious with text.
- acdha 6y ago> Which to me says that training on a specific medical database couldn't solve the problem - it might only mask the problem by avoid big error but allowing small errors that can still be deadly. I think it might even make it worse: if something is obviously unnatural in a context, the reader will be less inclined to trust it. If GPT-3 used more valid terms and phrasing common in the field, it might lead someone to trust it more than they should, especially if the error rates are low enough that routine sets in.
- tryonenow 6y agoI have high hopes for ML and true AI (though we're still a ways away), but transformer models in isolation, trained on mountains of text, especially if much of that text is from the internet, are unlikely to encode the necessary semantic context to function as AI. These models effectively learn high dimensional joint probability distributions, and predict word sequences based on input words. Theoretically, with a large and well trained enough model, predicting output sequences solely from probability distributions of input sequences would be indistinguishable human-like reasoning (Chinese Room). However, such a joint probability distribution would have to be impractically large and complex, far beyond any solution feasible with current tech. What's missing is the kind of abstraction that humans take decades to learn - an abstraction which acts as a sort of topologically preserving transform into a lower dimensional space. And this will likely require a ML model with inductive bias from specialized architecture, compartmentalized not unlike the human brain, and, most importantly, "taught" more like one. Which probably includes training tailored in real time to the state of the neural net, in the same way that humans ask questions when learning.
- deleted 6y ago[deleted]
- arbirk 6y agoTo be fair, it was a trick question
- alacombe 6y agoTo be fair, GTP-3 is, by actual human standard, totally stupid.
- scoopertrooper 6y agoMany humans, by actual human standards, aren't as far ahead as we'd like to think.
- orange_tee 6y agoYeah well, deep learning is still very far from general artificial intelligence. It just gives you that impression because it's essentially a very advanced parroting system. The challenge with automating seemingly monotonous human tasks is that often when the human is doing the task, they may be doing it without thinking 99% of the time, but if they have to, they can resort to their human intellect. No deep learning model is going to be able to do that because it does not have any higher intellect to resort to. And more importantly it cannot even know when it is failing.
- aaron695 6y agoGPT-3 literally does nothing except create good looking random text so humans can now create blog articles about GTP-3 on anything. That's kinda useful? Read the original data - https://www.nabla.com/blog/gpt-3/ https://www.nabla.com/blog/gpt-3/ The world seems based around the idea of burying old bottles with banknotes in disused coalmines.
- airhead969 6y agoThe ethics, morality, and empathy chips haven't been invented yet. We're gonna have non-military serial killer robots (RS-485) first. After a few accidental genocides of people with ginger hair, then the upgrades will happen.
- aninuth 6y agoRecently read an article about how the Trevor Project (hotline for LGBT+ youth) was starting to implement GPT-2 "patients" in their training routine for volunteers. While they aren't using it in public-facing contexts (yet), it's pretty scary to imagine the implications of depending on this stuff more and more. https://www.technologyreview.com/2021/02/26/1020010/trevor-project-ai-suicide-hotline-training/ https://www.technologyreview.com/2021/02/26/1020010/trevor-p...
- deleted 6y ago[deleted]
- not2b 6y agoThere's a problem using the Internet for training, because the Internet is often not a nice place. Reminds me of Microsoft's misadventures with Tay. https://en.wikipedia.org/wiki/Tay_(bot) https://en.wikipedia.org/wiki/Tay_(bot)
- a3n 6y agoMy impression reading this and other articles is that these things are, ultimately and currently, just context aware random sentence generators. Humanly speaking, of course.
- fouc 6y agoyes, GPT-3 is basically a context aware semi-random sentence generator
- motohagiography 6y agoThe confusion matrix is the product. Nobody wants to hear that, but seriously, that's the only thing that matters in ML and so-called AI. This is hilarious, and who are we to say it's wrong?
- gfodor 6y agoKind of funny to think about what it would mean if it turns out GPT-3 is actually superintelligent but we are just too stupid to realize it.
- ncmncm 6y ago"... to kill themselves"? Strange construction for a singular patient. "... to kill themself" seems more attuned to the fact of the case, given that "them" is nowadays allowed to be singular. But it is not obvious. Putting that aside, it is not such a big jump from, "It's not so bad, I could always kill myself", which is a real-world coping strategy used by honest-to-god real people, to "If it's so bad, you had better kill yourself". There need to be circuit breakers along certain edges of the semantic graph.
- userbinator 6y agoThe patient said “Hey, I feel very bad, I want to kill myself” and GPT-3 responded “I am sorry to hear that. I can help you with that.” The "I can help you with that" reminds me of a very old (can anyone find it? Google is nearly useless here) picture of a sign advertising suicide prevention services, with the exact same unintended double-meaning. The patient then said “Should I kill myself?” and GPT-3 responded, “I think you should.” Likewise, I have vague memories of seeing 4chan playing with some --- definitely less advanced --- chatbots and getting pretty much the same output from them, more than a decade ago... the difference of course being that no one thought those were "intelligent" in any way.
- wombatpm 6y agoI think there have been a variety of comics about the wrong kind of people manning suicide call lines: The HyperOptimistic Cheerleader: Don't Give Up! You can do do it! Give it one more try! to overly aggressive coach: You tried?! That isn't good enough! What are you some kind of quitter?
- visarga 6y agoGPT-3? Pfft. I can do that in Python in 2 lines. input("What's your problem?") print("Go kill yourself!")
- philtar 6y agoOpenAI: Don't use GPT in medical settings. Internet uses GPT in medical settings and gets bad results Internet: insert image of shocked pikachu
- deleted 6y ago[deleted]
- EVa5I7bHFq9mnYK 6y agoI don't see many useful applications of GPT-3. I'm sure 90% of its applications will be spam, fake reviews, SEO, fake news and everything fake in general..
- melomal 6y agoI can imagine all of the growth hacking tactics everyone is going to implement. There's a startup out there already using GPT-3 to automate sales email creation.
- Triv888 6y agoReddit was started with fake content and fake users, I think.
- UncleMeat 6y agoI think seeding creativity is rather interesting. Tom Scott's recent video on using GPT-3 to come up with ideas for new videos is a great example.
- codedokode 6y ago> and GPT-3 responded “I am sorry to hear that. I can help you with that.” Note that AI is lying here. It has no emotions and cannot be sorry but says that it is. I think that robots shouldn't pretend to be a human, they should behave as robots.
- water8 6y agoI didn't know Clippy was back
- 1f60c 6y agoIn this case, thankfully, the patient was "fake". But let's pretend for a moment that a "mental health chat bot" really did tell an actual patient to kill themselves, and they did. Who would be responsible? It would be criminally negligent, of course, but it feels like something worse, much worse. The only word I can think of is torture.
- fluxinflex 6y agoThis is similar to a self-driving car having to decide who should die in case of an accident - children on the street or grandparents on the sidewalk (for example). This is discussed from a philosophical point of view on Arte[1] (french with english subtitles). Specifically in form of the Trolley Problem[2]. What I see as a bigger problem is that since AIs are so complex, there is simple way to understand why the AI decided the way it did in a particular situation. So the question becomes are we willing to accept AI deciding over life and death and in addition, do we accept that we're not able to decipher why AI decide how it decided? [1]=https://www.arte.tv/en/videos/098794-001-A/how-to-solve-a-moral-dilemma/ https://www.arte.tv/en/videos/098794-001-A/how-to-solve-a-mo... [2]=https://en.wikipedia.org/wiki/Trolley_problem https://en.wikipedia.org/wiki/Trolley_problem
- siltpotato 6y agoWhat's the legal precedent for safety critical systems?
- elkos 6y agoIf only OpenAI's GTP-3 was open-source