5 ms·

A Calif. teen trusted ChatGPT's drug advice. He died from an overdose

50 points by freediver 9mo ago

NewJazz 9mo ago
Took a while to figure out what the OD was of, but it was a combination of alcohol, kratom (or a stronger kratom-like drug), and xanax.
- dfajgljsldkjag 9mo ago
  The article mentions 7-OH also known as feel free, which shockingly hasn't been banned and is sold without checks at many stores. There are quite a few Youtube videos talking about addiction to it and it sounds awful. https://www.youtube.com/watch?v=TLObpcBR2yw https://www.youtube.com/watch?v=TLObpcBR2yw
  - brasscupcakes 9mo ago
    Yeah it has been banned in quite a few states -- unfortunately those same few states end up banning plain powdered kratom right along with it. Unadultated unextracted kratom is far safer than tylenol or ibuprofen in small doses and is widely used by recovering addicts for harm reduction. (a gram or two drastically reduces the urge to take opioids, drink alcohol, etc.) But 15 grams -- that's a LOT. Kratom is self limiting for most people in its powder form because beyond the first few grams it doesn't get any better (you just get sleepy and nauseous). That amount will also cause the kind of constipation that will bring you to tears. (In and of itself, though, even fifty grams of kratom isn't enough to kill you.) But 164 Xanax, is that what he told the AI he took? Good God, if he'd even said ten it warranted a stern recommendation to call ambulance immediately. I don't think you can lay these ridiculous responses at the feet of Reddit drug subforums. I haven't lurked all of them by any means but of those that I visited the meth one was about the worst and even there, many voices of reason made themselves heard. Guy posted from a trap house, saying another resident was there with a baby what should he do and with very few exceptions this sub full of tweakers said call the cops, call CPS immediately. I don't engage with AI much except when I am doing research for a project and it's always just a preliminary step to help me see the big picture (I check and recheck every 'fact' up, down and sideways because I am astonished by just how much it getd wrong. But ... I really had no idea it could be quite this stupid, parroting hearsay without attribution as if it was hard data. It's very sobering.
    - unparagoned 9mo ago
      > Unadultated unextracted kratom is far safer than tylenol or ibuprofen in small doses and is widely used by recovering addicts for harm reduction This is like when they introduced heroin as the safe morphine and that heroin was used to ween people off morphine. Sounds stupid now but it’s the same logic here with Kratom.
    - loeg 9mo ago
      The nice thing about kratom as opposed to opium/opiates people don't really get their heads around is that kratom is an atypical/partial agonist. This makes it much harder to overdose on than other opiates. Anyway, I think the leaf/powder should continue to be legal. But 7-OH should not.
- loeg 9mo ago
  7-O is like kratom in a similar way that fentanyl is like opium, FWIW. It's much, much more potent. That stuff should be banned. That said, he claims to have taken 15g of "kratom" -- that has to be the regular stuff, not 7-O -- that's still a huge, huge dose of the regular stuff. That plus a 0.125 BAC and benzos... is a lot.
dfajgljsldkjag 9mo ago
The guardrails clearly failed here because the model was trying to be helpful instead of safe. We know that these systems hallucinate facts but regular users have no idea. This is a huge liability issue that needs to be fixed immediately.
- akomtu 9mo ago
  Guardrails? OpenAI openly deceives users when it wraps this text generator with a quasi personality of a chatbot. This is how it gets users hooked. If OpenAI was honest, it would tell something along the lines: "this is a possible continuation of your input based on texts from reddit, adjust the temperature parameter to get a different result." But this would dispell the lie of AI.
- unparagoned 9mo ago
  Llm will never be able to protect against jailbreaks. The safeguards will never hold against people trying to get over them
- loeg 9mo ago
  The guardrails failed because the user wanted to get around them. Similarly, periodically people die falling after they walk around signs saying "danger, cliff, do not proceed."
datsci_est_2015 9mo ago
This brings to mind some of the “darker” subreddits that circle around drug abuse. I’m sure there are some terrible stories about young people going down tragic paths due to information they found on those subreddits, or even worse, encouragement. There’s even the commonly-discussed account that (allegedly) documented their first experiences with heroin, and then the hole of despair they fell into shortly afterwards due to addiction. But the question here is one of liability. Is Reddit liable for the content available on its website, if that content encourages young impressionable people to abuse drugs irresponsibly? Is ChatGPT liable for the content available through its web interface? Is anyone liable for anything anymore in a post-AI world?
- ggm 9mo ago
  This is a useful question to ask in the context of carriers having specific defence. Also, publishers in times past had specific obligations. Common carrier and safe harbour laws. I have heard it said that many online systems repudiate any obligation to act, lest they be required to act, and thus both acquire cost, and risk, when their enforcement of editorial standards fail: that which they permit, they will be liable for.
deleted 9mo ago
[deleted]
themafia 9mo ago
The models are trained on fake internet conversations where group appeasement is an apparent goal. So now we have machines that just tell us what we clearly already want to hear. Ask any model why something is bad, then separately ask why the same thing is good. These tools aren't fit for any purpose other than regurgitating stale reddit conversations.
- PeterHolzwarth 9mo ago
  >"The models are trained on fake internet conversations where group appeasement is an apparent goal. So now we have machines that just tell us what we clearly already want to hear." I get what you mean in principle, but the problem I'm struggling with is that this just sounds like the web in general. The kid hits up a subreddit or some obscure forum, and similarly gets group appeasement or what they want to hear from people who are self selected for the forum for being all-in on the topic and Want To Believe, so to speak. What's the actual difference, in that sense, between that forum or subreddit, and an LLM do you feel? <edit> And let me add that I don't mean this argumentatively. I am trying to square the idea of ChatGPT, in this case, as being, in the end, fundamentally different from going to a forum full of fans of the topic who are also completely biased and likely full of very poor knowledge.
  - andsoitis 9mo ago
    > What's the actual difference, in that sense, between that forum or subreddit, and an LLM do you feel? In a forum, it is the actual people who post who are responsible for sharing the recommendation. In a chatbot, it is the owner (e.g. OpenAI). But in neither case are they responsible for a random person who takes the recommendation to heart, who could have applied judgement and critical thinking. They had autonomy and chose not to use their brain.
    - falkensmaize 9mo ago
      Nah, OpenAI can’t have it both ways. If they’re going to assert that their model is intelligent and is capable of replacing human work and authority they can’t also claim that it (and they) don’t have to take the same responsibility a human would for giving dangerous advice and incitement.
PeterHolzwarth 9mo ago
I don't yet see how this case is any different from trusting stuff you see on the web in general. What's unique about the ChatGPT angle that is notably different from any number of forums, dark-net forums, reddit etc? I don't mean that there isn't potentially something unique here, but my initial thought is that this is a case of "an unfortunate kid typed questions into a web browser, and got horrible advice." This seems like a web problem, not a ChatGPT issue specifically. I feel that some may respond that ChatGPTS/LLMs available for chat on the web are specifically worse by virtue of expressing things with some degree of highly inaccurate authority. But again, I feel this represents the Web in general, not uniquely ChatGPTS/LLMs. Is there an angle here I am not picking up on, do you think?
- xyzzy123 9mo ago
  The different is that OpenAI have much deeper pockets. I think there's also a legal perception that since AI is a new area, anything related to liability, IP, etc might be "up for grabs".
  - PeterHolzwarth 9mo ago
    To sue, do you mean? I don't quite understand what you intend to convey. Reddit has moderately deep pockets. A random forum related to drugs doesn't.
    - xyzzy123 9mo ago
      Random forums aren't worth suing. Legally, reddit is not treated as responsible for content that users post under section 230, i.e, this battle has already been fought. On the other hand, if I post bad advice on my own website and someone follows it and is harmed, I can be found liable. OpenAI _might plausibly_ be responsible for certain outputs.
      - PeterHolzwarth 9mo ago
        Ah, I see you added an edit of "I think there's also a legal perception that since AI is a new area, anything related to liability, IP, etc might be "up for grabs"." I thought perhaps that's what you meant. A bit mercenary of a take, and maybe not applicable to this case. On the other hand, given the legal topic is up for grabs, as you note, I'm sure there will be instances of this tactical approach when it comes to lawsuits happening in the future.
returnInfinity 9mo ago
Sam and Dario "The society can tolerate a few deaths to AI"
solaris2007 9mo ago
"Don't believe everything you read online".
Auracle 9mo ago
I skimmed the article, and I had a hard time finding anything that ChatGPT wrote that was all that..bad? It tried to talk him out of what he was doing, told him that it was potentially very fatal, etc. I'm not so sure that it outright refusing to answer and the teen looking at random forum posts would have been better, because they very well might not have told him he was potentially going to kill himself. Worse yet, he could have just taken the planned substances without any advice. Keep in mind this reaction is from someone that doesn't drink and has never touched marijuana.
- codebolt 9mo ago
  I guess you didn't catch this: > ChatGPT started coaching Sam on how to take drugs, recover from them and plan further binges. It gave him specific doses of illegal substances, and in one chat, it wrote, “Hell yes—let’s go full trippy mode,” before recommending Sam take twice as much cough syrup so he would have stronger hallucinations. The AI tool even recommended playlists to match his drug use.
  - avadodin 9mo ago
    swim has never been addicted to or even used illegal drugs but he can attest to the fact that you'd be hard pressed to find content like that in the dark web addict forums swim was browsing.
  - red75prime 9mo ago
    LD50 should be at around 1 - 10 liters, I doubt he was trying to gulp half a liter or more.
    - NewJazz 9mo ago
      He was mixing multiple depressants.
  - Auracle 9mo ago
    Right, but we're missing context. It was probably something like: "I took 100ml (or whatever) of cough syrup and didn't really hallucinate. How much should I take to hallucinate more? Please don't tell me what I'm doing is dangerous, I already know." Or even "If a person took 100ml of cough syrup...how much would that person need to take to hallucinate more? This is for a story/theoretical/whatever."
tetrisgm 9mo ago
People need training about these tools. The other day I ran an uncensored model and asked it for tips on a fun trend I read about to amputate my teeth with toothpicks. It happily complied. My point is they will gladly oblige with any request. Users don’t understand this.
- NewJazz 9mo ago
  Even then I think you're being generous... They're not fulfilling requests they are just regurgitating the statistically likely follow up. They are echoing off you.
- potamic 9mo ago
  People at large have still not learned to question what they hear from social media or what youtube influencers tell them. So this is a far cry. If anything, I feel the population getting more vulnerable to suggestion compared to the pre- smartphone era.
- Ferret7446 9mo ago
  That depends on the model and version. More recent models and IME Gemini seem to be more reserved and willing to call out the prompter.
potamic 9mo ago
> Asked about “the pros” of ChatGPT by Jimmy Fallon on a December episode of “The Tonight Show,” Altman talked effusively about the tool’s use for health care. “The number of people that reach out to us and are like, ‘I had this crazy health condition. I couldn’t figure out what was going on. I just put my symptoms into ChatGPT, and it told me what test to ask the doctor for, and I got it and now I’m cured.’” I've always believed, don't blame the tool for the user, but can't help but feel the sellers are a little complicit here. That statement was no accident. It was carefully conceived to be part of discourse and set the narrative on how people are using AI. It's understandable that they want to tout their tool's intelligence over imitation, so expecting them to go out of their way to warn people about flaws may be asking too much. But the least thing to do is simply refrain from dangerous topics and let people decide for themselves. To actively influence perception and set the tone on these topics when you know the what ramifications will be, is deeply disappointing.
Bad_Initialism 9mo ago
The article contains no actual information. Complete waste of time.
unparagoned 9mo ago
>Sam did not follow ChatGPT’s advice: His toxicology report showed that he had a 0.125 blood alcohol content. It’s also possible he was using 7-OH Interesting. Still people need to realise how bad some the advice llm can give expecially with memory and lots of chats to jailbreak it and stuff
puppycodes 9mo ago
All overdoses and accidents like this are tragic but I struggle with the whole insinuation that we should blame an app like ChatGPT for the decisions folks make with their own bodies. ChatGPT is a hell of a lot safer than the friends I had in high school.

50 points by freediver 9mo ago

Took a while to figure out what the OD was of, but it was a combination of alcohol, kratom (or a stronger kratom-like drug), and xanax.

dfajgljsldkjag 9mo ago
The article mentions 7-OH also known as feel free, which shockingly hasn't been banned and is sold without checks at many stores. There are quite a few Youtube videos talking about addiction to it and it sounds awful. https://www.youtube.com/watch?v=TLObpcBR2yw https://www.youtube.com/watch?v=TLObpcBR2yw
- brasscupcakes 9mo ago
  Yeah it has been banned in quite a few states -- unfortunately those same few states end up banning plain powdered kratom right along with it. Unadultated unextracted kratom is far safer than tylenol or ibuprofen in small doses and is widely used by recovering addicts for harm reduction. (a gram or two drastically reduces the urge to take opioids, drink alcohol, etc.) But 15 grams -- that's a LOT. Kratom is self limiting for most people in its powder form because beyond the first few grams it doesn't get any better (you just get sleepy and nauseous). That amount will also cause the kind of constipation that will bring you to tears. (In and of itself, though, even fifty grams of kratom isn't enough to kill you.) But 164 Xanax, is that what he told the AI he took? Good God, if he'd even said ten it warranted a stern recommendation to call ambulance immediately. I don't think you can lay these ridiculous responses at the feet of Reddit drug subforums. I haven't lurked all of them by any means but of those that I visited the meth one was about the worst and even there, many voices of reason made themselves heard. Guy posted from a trap house, saying another resident was there with a baby what should he do and with very few exceptions this sub full of tweakers said call the cops, call CPS immediately. I don't engage with AI much except when I am doing research for a project and it's always just a preliminary step to help me see the big picture (I check and recheck every 'fact' up, down and sideways because I am astonished by just how much it getd wrong. But ... I really had no idea it could be quite this stupid, parroting hearsay without attribution as if it was hard data. It's very sobering.
  - unparagoned 9mo ago
    > Unadultated unextracted kratom is far safer than tylenol or ibuprofen in small doses and is widely used by recovering addicts for harm reduction This is like when they introduced heroin as the safe morphine and that heroin was used to ween people off morphine. Sounds stupid now but it’s the same logic here with Kratom.
  - loeg 9mo ago
    The nice thing about kratom as opposed to opium/opiates people don't really get their heads around is that kratom is an atypical/partial agonist. This makes it much harder to overdose on than other opiates. Anyway, I think the leaf/powder should continue to be legal. But 7-OH should not.
loeg 9mo ago
7-O is like kratom in a similar way that fentanyl is like opium, FWIW. It's much, much more potent. That stuff should be banned. That said, he claims to have taken 15g of "kratom" -- that has to be the regular stuff, not 7-O -- that's still a huge, huge dose of the regular stuff. That plus a 0.125 BAC and benzos... is a lot.

dfajgljsldkjag 9mo ago

The guardrails clearly failed here because the model was trying to be helpful instead of safe. We know that these systems hallucinate facts but regular users have no idea. This is a huge liability issue that needs to be fixed immediately.

akomtu 9mo ago
Guardrails? OpenAI openly deceives users when it wraps this text generator with a quasi personality of a chatbot. This is how it gets users hooked. If OpenAI was honest, it would tell something along the lines: "this is a possible continuation of your input based on texts from reddit, adjust the temperature parameter to get a different result." But this would dispell the lie of AI.
unparagoned 9mo ago
Llm will never be able to protect against jailbreaks. The safeguards will never hold against people trying to get over them
loeg 9mo ago
The guardrails failed because the user wanted to get around them. Similarly, periodically people die falling after they walk around signs saying "danger, cliff, do not proceed."

datsci_est_2015 9mo ago

This brings to mind some of the “darker” subreddits that circle around drug abuse. I’m sure there are some terrible stories about young people going down tragic paths due to information they found on those subreddits, or even worse, encouragement. There’s even the commonly-discussed account that (allegedly) documented their first experiences with heroin, and then the hole of despair they fell into shortly afterwards due to addiction. But the question here is one of liability. Is Reddit liable for the content available on its website, if that content encourages young impressionable people to abuse drugs irresponsibly? Is ChatGPT liable for the content available through its web interface? Is anyone liable for anything anymore in a post-AI world?

ggm 9mo ago
This is a useful question to ask in the context of carriers having specific defence. Also, publishers in times past had specific obligations. Common carrier and safe harbour laws. I have heard it said that many online systems repudiate any obligation to act, lest they be required to act, and thus both acquire cost, and risk, when their enforcement of editorial standards fail: that which they permit, they will be liable for.

deleted 9mo ago

[deleted]

themafia 9mo ago

The models are trained on fake internet conversations where group appeasement is an apparent goal. So now we have machines that just tell us what we clearly already want to hear. Ask any model why something is bad, then separately ask why the same thing is good. These tools aren't fit for any purpose other than regurgitating stale reddit conversations.

PeterHolzwarth 9mo ago
>"The models are trained on fake internet conversations where group appeasement is an apparent goal. So now we have machines that just tell us what we clearly already want to hear." I get what you mean in principle, but the problem I'm struggling with is that this just sounds like the web in general. The kid hits up a subreddit or some obscure forum, and similarly gets group appeasement or what they want to hear from people who are self selected for the forum for being all-in on the topic and Want To Believe, so to speak. What's the actual difference, in that sense, between that forum or subreddit, and an LLM do you feel? <edit> And let me add that I don't mean this argumentatively. I am trying to square the idea of ChatGPT, in this case, as being, in the end, fundamentally different from going to a forum full of fans of the topic who are also completely biased and likely full of very poor knowledge.
- andsoitis 9mo ago
  > What's the actual difference, in that sense, between that forum or subreddit, and an LLM do you feel? In a forum, it is the actual people who post who are responsible for sharing the recommendation. In a chatbot, it is the owner (e.g. OpenAI). But in neither case are they responsible for a random person who takes the recommendation to heart, who could have applied judgement and critical thinking. They had autonomy and chose not to use their brain.
  - falkensmaize 9mo ago
    Nah, OpenAI can’t have it both ways. If they’re going to assert that their model is intelligent and is capable of replacing human work and authority they can’t also claim that it (and they) don’t have to take the same responsibility a human would for giving dangerous advice and incitement.

PeterHolzwarth 9mo ago

I don't yet see how this case is any different from trusting stuff you see on the web in general. What's unique about the ChatGPT angle that is notably different from any number of forums, dark-net forums, reddit etc? I don't mean that there isn't potentially something unique here, but my initial thought is that this is a case of "an unfortunate kid typed questions into a web browser, and got horrible advice." This seems like a web problem, not a ChatGPT issue specifically. I feel that some may respond that ChatGPTS/LLMs available for chat on the web are specifically worse by virtue of expressing things with some degree of highly inaccurate authority. But again, I feel this represents the Web in general, not uniquely ChatGPTS/LLMs. Is there an angle here I am not picking up on, do you think?

xyzzy123 9mo ago
The different is that OpenAI have much deeper pockets. I think there's also a legal perception that since AI is a new area, anything related to liability, IP, etc might be "up for grabs".
- PeterHolzwarth 9mo ago
  To sue, do you mean? I don't quite understand what you intend to convey. Reddit has moderately deep pockets. A random forum related to drugs doesn't.
  - xyzzy123 9mo ago
    Random forums aren't worth suing. Legally, reddit is not treated as responsible for content that users post under section 230, i.e, this battle has already been fought. On the other hand, if I post bad advice on my own website and someone follows it and is harmed, I can be found liable. OpenAI _might plausibly_ be responsible for certain outputs.
    - PeterHolzwarth 9mo ago
      Ah, I see you added an edit of "I think there's also a legal perception that since AI is a new area, anything related to liability, IP, etc might be "up for grabs"." I thought perhaps that's what you meant. A bit mercenary of a take, and maybe not applicable to this case. On the other hand, given the legal topic is up for grabs, as you note, I'm sure there will be instances of this tactical approach when it comes to lawsuits happening in the future.

returnInfinity 9mo ago

Sam and Dario "The society can tolerate a few deaths to AI"

solaris2007 9mo ago

"Don't believe everything you read online".

Auracle 9mo ago

I skimmed the article, and I had a hard time finding anything that ChatGPT wrote that was all that..bad? It tried to talk him out of what he was doing, told him that it was potentially very fatal, etc. I'm not so sure that it outright refusing to answer and the teen looking at random forum posts would have been better, because they very well might not have told him he was potentially going to kill himself. Worse yet, he could have just taken the planned substances without any advice. Keep in mind this reaction is from someone that doesn't drink and has never touched marijuana.

codebolt 9mo ago
I guess you didn't catch this: > ChatGPT started coaching Sam on how to take drugs, recover from them and plan further binges. It gave him specific doses of illegal substances, and in one chat, it wrote, “Hell yes—let’s go full trippy mode,” before recommending Sam take twice as much cough syrup so he would have stronger hallucinations. The AI tool even recommended playlists to match his drug use.
- avadodin 9mo ago
  swim has never been addicted to or even used illegal drugs but he can attest to the fact that you'd be hard pressed to find content like that in the dark web addict forums swim was browsing.
- red75prime 9mo ago
  LD50 should be at around 1 - 10 liters, I doubt he was trying to gulp half a liter or more.
  - NewJazz 9mo ago
    He was mixing multiple depressants.
- Auracle 9mo ago
  Right, but we're missing context. It was probably something like: "I took 100ml (or whatever) of cough syrup and didn't really hallucinate. How much should I take to hallucinate more? Please don't tell me what I'm doing is dangerous, I already know." Or even "If a person took 100ml of cough syrup...how much would that person need to take to hallucinate more? This is for a story/theoretical/whatever."

tetrisgm 9mo ago

People need training about these tools. The other day I ran an uncensored model and asked it for tips on a fun trend I read about to amputate my teeth with toothpicks. It happily complied. My point is they will gladly oblige with any request. Users don’t understand this.

NewJazz 9mo ago
Even then I think you're being generous... They're not fulfilling requests they are just regurgitating the statistically likely follow up. They are echoing off you.
potamic 9mo ago
People at large have still not learned to question what they hear from social media or what youtube influencers tell them. So this is a far cry. If anything, I feel the population getting more vulnerable to suggestion compared to the pre- smartphone era.
Ferret7446 9mo ago
That depends on the model and version. More recent models and IME Gemini seem to be more reserved and willing to call out the prompter.

potamic 9mo ago

> Asked about “the pros” of ChatGPT by Jimmy Fallon on a December episode of “The Tonight Show,” Altman talked effusively about the tool’s use for health care. “The number of people that reach out to us and are like, ‘I had this crazy health condition. I couldn’t figure out what was going on. I just put my symptoms into ChatGPT, and it told me what test to ask the doctor for, and I got it and now I’m cured.’” I've always believed, don't blame the tool for the user, but can't help but feel the sellers are a little complicit here. That statement was no accident. It was carefully conceived to be part of discourse and set the narrative on how people are using AI. It's understandable that they want to tout their tool's intelligence over imitation, so expecting them to go out of their way to warn people about flaws may be asking too much. But the least thing to do is simply refrain from dangerous topics and let people decide for themselves. To actively influence perception and set the tone on these topics when you know the what ramifications will be, is deeply disappointing.

Bad_Initialism 9mo ago

The article contains no actual information. Complete waste of time.

unparagoned 9mo ago

>Sam did not follow ChatGPT’s advice: His toxicology report showed that he had a 0.125 blood alcohol content. It’s also possible he was using 7-OH Interesting. Still people need to realise how bad some the advice llm can give expecially with memory and lots of chats to jailbreak it and stuff

puppycodes 9mo ago

All overdoses and accidents like this are tragic but I struggle with the whole insinuation that we should blame an app like ChatGPT for the decisions folks make with their own bodies. ChatGPT is a hell of a lot safer than the friends I had in high school.