6 ms·

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguide

by sumeno 1y ago

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.

MBCook 1y ago
How many of these cases exist in the other direction? Where AI chatbots have actively harmed people’s mental health, including possible to the point of self destructive behaviors or self harm? A single positive outcome is not enough to judge the technology beneficial, let alone safe.
- j_timberlake 1y ago
  This is called the "Man bites dog" bias. The many people who don't commit suicide because an AI confidant helped them out are never ever gonna make the news. Meanwhile the opposite cases are "TODAY'S TOP HEADLINE" and that's what people discuss.
- throwawaybob420 1y ago
  idk dude if your technology encourages a teenager to kill itself and prevents him from alerting his parents via a cry for help, I don’t care how “beneficial” it is.
  - MBCook 1y ago
    I agree. If there was one death for 1 million saves, maybe. Instead, this just came up in my feed: https://arstechnica.com/tech-policy/2025/08/chatgpt-helped-teen-plan-suicide-after-safeguards-failed-openai-admits/ https://arstechnica.com/tech-policy/2025/08/chatgpt-helped-t...
    - rideontime 1y ago
      This is the same case that is being discussed, and your comment up-thread does not demonstrate awareness that you are, in fact, agreeing with the parent comment that you replied to. I get the impression that you read only the headline, not the article, and assumed it was a story about someone using ChatGPT for therapy and gaining a positive outcome.
      - MBCook 1y ago
        I did! Because I can’t see past the paywall. I can’t even read the first paragraph. So the headline is the only context I have.
        rideontime 1y ago
        I would advise you to gather more context before commenting in the future.
        latexr 1y ago
        A link to bypass the paywall has been posted several hours before your comment, and currently sits at the top. https://news.ycombinator.com/item?id=45027043 https://news.ycombinator.com/item?id=45027043 I recommend you get in the habit of searching for those. They are often posted, guaranteed on popular stories. Commenting without context does not make for good discussion.
    - mvdtnz 1y ago
      What on Earth? You're posting an article about the same thing we're already discussing. If you want to contribute to the conversation you owe it to the people who are taking time out of their day to engage with you to read the material under discussion.
  - threatofrain 1y ago
    Although I don't believe current technology is ready for talk therapy, I'd say that anti-depressants can also cause suicidal thoughts and feelings. Judging the efficacy of medical technology can't be done on this kind of moral absolutism.
    - AIPedant 1y ago
      I think it's fine to be "morally absolutist" when it's non-medical technology, developed with zero input from federal regulators, yet being misused and misleadingly marketed for medical purposes.
    - podgietaru 1y ago
      The suicidal ideation of Antidepressants is a well communicated side effect. Antidepressants are prescribed by trained medical professionals who will tell you, encourage you to tell them if these side effects occur, and will encourage you to stop the medication if it does occur. It's almost as if we've built systems around this stuff for a reason.
      - Denatonium 1y ago
        In practice, they'll just prescribe a higher dose when that happens, thus worsening the problem. I'm not defending the use of AI chatbots, but you'd be hard-pressed to come up with a worse solution for depression than the medical system.
        podgietaru 1y ago
        Not my experience at all. The Psych that prescribed me antidepressants was _incredibly_ diligent. Including with side effects that affected my day to day life like loss of Libido. We spent a long time finding something, but when we did it worked exceptionally well. We absolutely did not just increase the dose. And I'm almost certain the literature for this would NOT recommend an increase of dosage if the side effect was increased suicidality. The demonisation of medication needs to stop. It is an important tool in the toolbelt for depression. It is not the end of the journey, but it makes that journey much easier to walk.
  - npteljes 1y ago
    You might not care personally, but this isn't how we do anything, because we wouldn't have anything in the world at all. Different things harm and kill people all the time, and many of them have barely any use than harmful activity, yet they are still active parts of our lives. I understand the emotional impact of what happened in this case, but there is not much to discuss if we just reject everything outright.
- kayodelycaon 1y ago
  It’s way more common than you think. I’m in a bubble of anti-AI people and we can see people we know going down that road. My family (different bubble) knows people. Every group of people I know knows somebody doing this. For context, my friends and family are in the northern Midwest. Average people, not early adopters of new technology.
MajimasEyepatch 1y ago
Exactly right. It's totally plausible that someone could build a mental health chatbot that results in better outcomes than people who receive no support, but that's a hypothesis that can and should be tested and subject to strict ethical oversight.
UltraSane 1y ago
I don't know if it counts as therapy or not but I find the ability to have intelligent (seeming?) conversations with Claude about the most incredibly obscure topics to be very pleasant.
- hattmall 1y ago
  But do you really feel you are conversing? I could never get that feeling. It's not a conversation to me it's just like an on-demand book that might be wrong. Not saying I don't use them to attempt to get information, but it certainly doesn't have a feeling than doing anything other than getting information out of a computer.
  - UltraSane 1y ago
    "But do you really feel you are conversing?" Yes. For topics with lots of training data like physics Claude is VERY human sounding. I've had very interesting conversations with Claude Opus about the Boltzmann brain issue and how I feel that the conventional wisdom ignores the low probability of a BBrain having a spatially and temporally consistent set of memories and how the fact that brains existing in a universe that automatically creates consistent memories means the probability of us being Boltzmann brains is very low. Since even if a Boltzmann brain pops into existence its memory will be most likely completely random and completely insane/insensate. There aren't a lot of people who want to talk about Boltzmann brains.
    - furyofantares 1y ago
      It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.
      - UltraSane 1y ago
        "It sounds like you're mostly just talking to yourself" No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong. Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamentally new way to index data.
npteljes 1y ago
Yeah, I was one such person, but I might give up on this ultimately. If I will, I will do so for CYA reasons, not because I think it's a bad thing overall. In this current case, the outcome is horrible, and the answers that ChatGPT provided were inexcusable. But looking at a bigger picture, how much of a better chance does a person have by everyone telling them to "go to therapy" or to "talk to others" and such? What others? Searching "online therapy", BetterHelp is the second result. BetterHelp doesn't exactly have a good reputation online, but still, their influence is widespread. Licensed therapists can also be bad actors. There is no general "good thing" that is tried and true for every particular case of human mental health, but even letting that go, the position is abused just as any other authority / power position is, with many bad therapists out there. Not to mention the other people who pose as (mental) health experts, life coaches, and such. Or the people who recruit for a cult. Frankly, even in the face of this horrible event, I'm not convinced that AI in general fares that much lower than the sum of the people who offer a recipe for a better life, skills, company, camaraderie. Rather, I feel like that AI is in a situation like the self-driving cars are, where we expect the new thing to be 110%, even though we know that the old thing is far for perfect. I do think that OpenAI is liable though, and rightfully so. Their service has a lot of power to influence, clearly outlined in the tragedy that is shown in the article. And so, they also have a lot of responsibility to reign that in. If they were a forum where the teen was pushed to suicide, police could go after the forum participants, moderators, admins. But in case of OpenAI, there is no such person, the service itself is the thing. So the one liable must be the company that provides the service.
staticman2 1y ago
There's no indication the kid asked ChatGPT to act as a therapist. Unless people are claiming any prompt is better than no therapy I don't think your framing is fair.

by sumeno 1y ago

MBCook 1y ago

How many of these cases exist in the other direction? Where AI chatbots have actively harmed people’s mental health, including possible to the point of self destructive behaviors or self harm? A single positive outcome is not enough to judge the technology beneficial, let alone safe.

j_timberlake 1y ago
This is called the "Man bites dog" bias. The many people who don't commit suicide because an AI confidant helped them out are never ever gonna make the news. Meanwhile the opposite cases are "TODAY'S TOP HEADLINE" and that's what people discuss.
throwawaybob420 1y ago
idk dude if your technology encourages a teenager to kill itself and prevents him from alerting his parents via a cry for help, I don’t care how “beneficial” it is.
- MBCook 1y ago
  I agree. If there was one death for 1 million saves, maybe. Instead, this just came up in my feed: https://arstechnica.com/tech-policy/2025/08/chatgpt-helped-teen-plan-suicide-after-safeguards-failed-openai-admits/ https://arstechnica.com/tech-policy/2025/08/chatgpt-helped-t...
  - rideontime 1y ago
    This is the same case that is being discussed, and your comment up-thread does not demonstrate awareness that you are, in fact, agreeing with the parent comment that you replied to. I get the impression that you read only the headline, not the article, and assumed it was a story about someone using ChatGPT for therapy and gaining a positive outcome.
    - MBCook 1y ago
      I did! Because I can’t see past the paywall. I can’t even read the first paragraph. So the headline is the only context I have.
      - rideontime 1y ago
        I would advise you to gather more context before commenting in the future.
      - latexr 1y ago
        A link to bypass the paywall has been posted several hours before your comment, and currently sits at the top. https://news.ycombinator.com/item?id=45027043 https://news.ycombinator.com/item?id=45027043 I recommend you get in the habit of searching for those. They are often posted, guaranteed on popular stories. Commenting without context does not make for good discussion.
  - mvdtnz 1y ago
    What on Earth? You're posting an article about the same thing we're already discussing. If you want to contribute to the conversation you owe it to the people who are taking time out of their day to engage with you to read the material under discussion.
- threatofrain 1y ago
  Although I don't believe current technology is ready for talk therapy, I'd say that anti-depressants can also cause suicidal thoughts and feelings. Judging the efficacy of medical technology can't be done on this kind of moral absolutism.
  - AIPedant 1y ago
    I think it's fine to be "morally absolutist" when it's non-medical technology, developed with zero input from federal regulators, yet being misused and misleadingly marketed for medical purposes.
  - podgietaru 1y ago
    The suicidal ideation of Antidepressants is a well communicated side effect. Antidepressants are prescribed by trained medical professionals who will tell you, encourage you to tell them if these side effects occur, and will encourage you to stop the medication if it does occur. It's almost as if we've built systems around this stuff for a reason.
    - Denatonium 1y ago
      In practice, they'll just prescribe a higher dose when that happens, thus worsening the problem. I'm not defending the use of AI chatbots, but you'd be hard-pressed to come up with a worse solution for depression than the medical system.
      - podgietaru 1y ago
        Not my experience at all. The Psych that prescribed me antidepressants was _incredibly_ diligent. Including with side effects that affected my day to day life like loss of Libido. We spent a long time finding something, but when we did it worked exceptionally well. We absolutely did not just increase the dose. And I'm almost certain the literature for this would NOT recommend an increase of dosage if the side effect was increased suicidality. The demonisation of medication needs to stop. It is an important tool in the toolbelt for depression. It is not the end of the journey, but it makes that journey much easier to walk.
- npteljes 1y ago
  You might not care personally, but this isn't how we do anything, because we wouldn't have anything in the world at all. Different things harm and kill people all the time, and many of them have barely any use than harmful activity, yet they are still active parts of our lives. I understand the emotional impact of what happened in this case, but there is not much to discuss if we just reject everything outright.
kayodelycaon 1y ago
It’s way more common than you think. I’m in a bubble of anti-AI people and we can see people we know going down that road. My family (different bubble) knows people. Every group of people I know knows somebody doing this. For context, my friends and family are in the northern Midwest. Average people, not early adopters of new technology.

MajimasEyepatch 1y ago

Exactly right. It's totally plausible that someone could build a mental health chatbot that results in better outcomes than people who receive no support, but that's a hypothesis that can and should be tested and subject to strict ethical oversight.

UltraSane 1y ago

I don't know if it counts as therapy or not but I find the ability to have intelligent (seeming?) conversations with Claude about the most incredibly obscure topics to be very pleasant.

hattmall 1y ago
But do you really feel you are conversing? I could never get that feeling. It's not a conversation to me it's just like an on-demand book that might be wrong. Not saying I don't use them to attempt to get information, but it certainly doesn't have a feeling than doing anything other than getting information out of a computer.
- UltraSane 1y ago
  "But do you really feel you are conversing?" Yes. For topics with lots of training data like physics Claude is VERY human sounding. I've had very interesting conversations with Claude Opus about the Boltzmann brain issue and how I feel that the conventional wisdom ignores the low probability of a BBrain having a spatially and temporally consistent set of memories and how the fact that brains existing in a universe that automatically creates consistent memories means the probability of us being Boltzmann brains is very low. Since even if a Boltzmann brain pops into existence its memory will be most likely completely random and completely insane/insensate. There aren't a lot of people who want to talk about Boltzmann brains.
  - furyofantares 1y ago
    It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.
    - UltraSane 1y ago
      "It sounds like you're mostly just talking to yourself" No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong. Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamentally new way to index data.

npteljes 1y ago

Yeah, I was one such person, but I might give up on this ultimately. If I will, I will do so for CYA reasons, not because I think it's a bad thing overall. In this current case, the outcome is horrible, and the answers that ChatGPT provided were inexcusable. But looking at a bigger picture, how much of a better chance does a person have by everyone telling them to "go to therapy" or to "talk to others" and such? What others? Searching "online therapy", BetterHelp is the second result. BetterHelp doesn't exactly have a good reputation online, but still, their influence is widespread. Licensed therapists can also be bad actors. There is no general "good thing" that is tried and true for every particular case of human mental health, but even letting that go, the position is abused just as any other authority / power position is, with many bad therapists out there. Not to mention the other people who pose as (mental) health experts, life coaches, and such. Or the people who recruit for a cult. Frankly, even in the face of this horrible event, I'm not convinced that AI in general fares that much lower than the sum of the people who offer a recipe for a better life, skills, company, camaraderie. Rather, I feel like that AI is in a situation like the self-driving cars are, where we expect the new thing to be 110%, even though we know that the old thing is far for perfect. I do think that OpenAI is liable though, and rightfully so. Their service has a lot of power to influence, clearly outlined in the tragedy that is shown in the article. And so, they also have a lot of responsibility to reign that in. If they were a forum where the teen was pushed to suicide, police could go after the forum participants, moderators, admins. But in case of OpenAI, there is no such person, the service itself is the thing. So the one liable must be the company that provides the service.

staticman2 1y ago

There's no indication the kid asked ChatGPT to act as a therapist. Unless people are claiming any prompt is better than no therapy I don't think your framing is fair.