12 ms·

If you haven't read the article (or even if you have but didn't click on outgoing links twice) the NYT story about how ChatGPT convinced a suicidal teen not to

If you haven't read the article (or even if you have but didn't click on outgoing links twice) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me,” Adam wrote at the end of March. > “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.” I am acutely aware that there's not enough psychologists out there but a sycophant bot is not the answer. One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing. [1] https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...

FloorEgg 11mo ago
What version of chatGPT was this? That sounds like a gpt-3.5 kind of response. I wonder how the latest models would fare in these kinds of conversations.
Al-Khwarizmi 11mo ago
We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa (not a social scientist so I have no idea what the methodology would look like, but if should be doable... or if it currently isn't, we should find the way).
- computably 11mo ago
  > Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa That is the literal opposite of how medical treatment is regulated. Treatments should be tested and studied before availability to the general public. It's irresponsible in the extreme to suggest this.
- grey-area 11mo ago
  A word generator with no intelligence or understanding based on the contents of the internet should not be allowed near suicidal teens, nor should it attempt to offer advice of any kind. This is basic common sense. Add in the commercial incentives of 'Open'AI to promote usage for anything and everything and you have a toxic mix.
  - butlike 11mo ago
    Yet, if you ask the word generator to generate words in the form of advice, like any machine or code, it will do exactly what you tell it to do. The fact people are asking implies a lack of common sense by your definition. Sertraline can increase suicidal thoughts in teens. Should anti-depressants not be allowed near suicidal/depressed teens?
    - grey-area 11mo ago
      Should anti-depressants not be allowed near suicidal/depressed teens? Well certainly not without careful monitoring and medical advice, no of course not!
  - Al-Khwarizmi 11mo ago
    Supposing that the advice it provides does more good than harm, why? What's the objective reason? If it can save lives, who cares if the advice is based on intelligence and understanding or on regurgitating internet content?
    - bayindirh 11mo ago
      > Supposing that the advice it provides does more good than harm, why? Because a human, esp. a confused and depressive human being is a complex thing. Much more complex than a stable, healthy human. Words encouraging a healthy person can break a depressed person further. Statistically positive words can deepen wounds, and push people more to the edge. Dark corners of human nature is twisted, hard to navigate and full of distortions. Simple words don't and can't help. Humans are not machines, brains are not mathematical formulae. We're not deterministic. We need to leave this fantasy behind.
- bayindirh 11mo ago
  Maybe it's causing even more deaths than we know, and these doesn't make the news either? If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news.
  - Al-Khwarizmi 11mo ago
    > Maybe it's causing even more deaths than we know, and these doesn't make the news either? Of course, and that's part of why I say that we need to measure the impact. It could be net positive or negative, we won't know if we don't find out. > If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news. I'm not advocating for not improving security, I'm arguing against a comment that said that "ChatGPT should be nowhere near anyone dealing with psychological issues", because it can cause death. Following your analogy, cars objectively cause deaths (and not only of people with psychological issues, but of people in general) and we don't say that "they should be nowhere near a person". We improve their safety even though zero deaths is probably impossible, which we accept because they are useful. This is a big-picture approach.
- jrflowers 11mo ago
  [flagged]
  - david-gpu 11mo ago
    They didn't say it was impossible, or that we should do nothing. Learn how to have a constructive dialogue, please.
    - jrflowers 11mo ago
      You make a good point. While they absolutely and unequivocally said that it is currently impossible to tell whether the suicides are bad or not, they also sort of wondered aloud if in the future we might be able to develop a methodology to determine whether the suicides are bad or not. This is an important distinction becau
- alex-moon 11mo ago
  I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. When the stakes are life or death, as they are with someone who is suicidal, that is a good example of a time when 80% isn't good enough. In such cases, where a new approach offers to replace an existing approach, the burden of proof is on the challenger, not the incumbent. This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed tests. You understand then, hopefully, why your comments here are dangerous...? I have no doubt you have no malicious intent here - you're right that these decisions need to be based on data - but you're not taking into account that the (potentially extremely harmful) challenger already has a foothold in the field. I know that you will want to hear this from experts in the "relevant field" rather than myself, so here is a write-up from Stanford on the subject: https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in...
  - butlike 11mo ago
    What are you talking about? I can grow food myself, and I can build a car from scratch and take it on the highway. Are there repercussions? Sure, but nothing inherently stops me from doing it. The problem here is there's no measurable "win condition" for when a person gets good information that helps them. They remain alive, which was their previous state. This is hard to measure. Now, should people be able to google their symptoms and try and help themselves? This dovetails into a deeper philosophical discussion, but I'm not entirely convinced "seek professional help" is ALWAYS the answer. ALWAYS and NEVER are _very_ long timeframes, and we should be careful when using them.
  - RobRivera 11mo ago
    I would take it a step back and posit simply, "why does a human require certification to practice therapy, and a computer program does not?" There should be some liability for malpractrice even if generated by an llm
  - 11mo ago
- toofy 11mo ago
  people are overcomplicating this, the big picture is simple af: if a therapist was ever found to have said to this to a suicidal person, they would be immediately stripped of their license and maybe jailed.
  - cal85 11mo ago
    True. But it feels like a fairer comparison would be with a huge healthcare company that failed to vet one of its therapists properly, so a crazy pro-suicide therapist slipped through the net. Would we petition to shut down the whole company for this rare event? I suppose it would depend on whether the company could demonstrate what it is doing to ensure it doesn’t happen again.
    - wat10000 11mo ago
      Maybe you shouldn't shut down OpenAI over this. But each instance of a particular ChatGPT model is the same as all the others. This is like a company that has a magical superhuman therapist that can see a million patients a day. If they're found to be encouraging suicide, then they need to be stopped from providing therapy. The fact that this is the company's only source of revenue might mean that the company has to shut down over this, but that's just a consequence of putting all your eggs in one basket.
  - terminalshort 11mo ago
    But you would have to be a therapist. If a suicidal person went up to a stranger and started a conversation, there would be no consequences. That's more analogous to ChatGPT.
- ml-anon 11mo ago
  This entire comment section is full of wide eyed nonsense like this. It’s honestly frightening that we are even humoring this point of view.
  - nihzm 11mo ago
    Since as you say this utilitarian view is rather common, perhaps it would good to show _why_ this is problematic by presenting a counterargument. The basic premise under GP's statements is that although not perfect, we should use the technology in such a way that it maximizes the well being of the largest number of people, even if comes at the expense of a few. But therein lies a problem: we cannot really measure well being (or utility). This becomes obvious if you look at individuals instead of the aggregate: imagine LLM therapy becomes widespread and a famous high profile person and your (not famous) daughter end up in "the few" for which LLM therapy goes terribly wrong and commit suicide. The loss of the famous person will cause thousands (perhaps millions) people to be a bit sad, and the loss of your daughter will cause you unimaginable pain. Which one is greater? Can they even be be compared? And how many people with a successful LLM therapy are enough to compensate for either one? Unmeasurable well-being then makes these moral calculations at best inexact and at worst completely meaningless. And if they are truly meaningless, how can they inform your LLM therapy policy decisions? Suppose for the sake of the argument we accept the above, and there is a way to measure well being. Then would it be just? Justice is a fuzzy concept, but imagine we reverse the example above: many people lose their lives because of bad LLM therapy, but one very famous person in the entertainment industry is saved by LLM therapy. Let's suppose then that this famous persons' well being plus the millions of spectators' improved well-being (through their entertainment) is worth enough to compensate the people who died. This means saving a famous funny person justifies the death of many. This does not feel just, does it? There is a vast amount of literature on this topic (criticisms of utilitarianism).
    - ml-anon 11mo ago
      This is either incredible satire or you’re a lunatic.
      - 11mo ago
- computerthings 11mo ago
  [dead]
- hitarpetar 11mo ago
  > someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa we already have an approval process for medical interventions. are you suggesting the government shut ChatGPT down until the FDA can investigate it's use for therapy? because if so I can get behind that
- wat10000 11mo ago
  Our standard approach for new medical treatments is to require proof of safety and efficacy before it's made available to the general public. This is because it's very, very easy for promising-looking treatments to end up being harmful. "Before declaring that it shouldn't be near anyone with psychological issues" is backwards. Before providing it to people with psychological issues, someone should study whether the positive impact is greater than the negative. Trouble is, this is such a generalized tool that it's very hard to do that.
- wfleming 11mo ago
  If a therapist helped 99/100 patients but tacitly encouraged the 100th to commit suicide* they would still lose their license. * ignoring the case of ethical assisted suicide for reasons of terminal illness and such, which doesn’t seem relevant to the case discussed here.
- FuckButtons 11mo ago
  Let’s maybe not give the benefit of the doubt to the startup which has shown itself to have the moral scruples of vault-tec just because what they’re doing might work out fine for some of the people they’re experimenting on.
bloqs 11mo ago
im willing to bet that it reduces them at a statistical level. knee jerk emotional reaction to a hallucinaton isnt the way forward with these things
torben-friis 11mo ago
>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Is that a debate worth having though? If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures. Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.
- probably_wrong 11mo ago
  > Is that a debate worth having though? Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions. As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop." [1] https://www.bbc.com/news/articles/cy5prvgw0r1o https://www.bbc.com/news/articles/cy5prvgw0r1o
  - arowthway 11mo ago
    Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
    - latexr 11mo ago
      > Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good. But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders. https://archive.ph/2025.08.08-145022/https://www.404media.co/guy-gives-himself-19th-century-psychiatric-illness-after-consulting-with-chatgpt/ https://archive.ph/2025.08.08-145022/https://www.404media.co... > From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy. From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.
p1dda 11mo ago
Exactly, any company that offers chatbots to the public should do what Google did regarding suicide searches, remove harmful websites and provide info how to contact mental health professionals. Anything else would be corporate suicide (pun not intended).
b3lvedere 11mo ago
"One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing." And how would a layman know the difference? If i desperately need help with mental item x and i have no clue how to get help, am very very ashamed for even mentioning to ask for help about mental item x or there are actually no resources available, i will turn to anything else than nothing. Because item x still exists and is making me suffer 24/7. At least the bot pretends to listen, some humans cannot even do that.
- probably_wrong 11mo ago
  I think you're being too generous to the idea that it could help without any evidence. If we assume that there's therapeutic value to bringing your problems out then a diary is a better tool. And if we believe that it's the feedback what's helping, well, we have cases of ChatGPT encouraging people's psychosis. We know that a layman often doesn't know the difference between what's helpful and what isn't - that's why loving relatives end up often enabling people's addictions thinking they're helping. But I'd argue that a system that confidently gives mediocre feedback at best and actively psychotic at worst is not a system that should be encouraged simply because it's cheap. I also wanted to snarkily write "even a dog would be better", but the more I thought about it the more I realized that yes, a dog would probably be a solid alternative.
  - b3lvedere 11mo ago
    I understand your explanation. But it might be quite tough for someone in a certain mental state to make rational decision, especially when they get the feeling there will never be help. Plenty of examples where people try and stay sane by talking to inert objects in desperation.
atleastoptimal 11mo ago
OpenAI tried to get rid of the excessively sycophantic model (4o) but there was a massive backlash. They eventually relented and kept it as a model offering in ChatGPT. OpenAI certainly has made mistakes with its rollouts in the past, but it is effectively impossible to keep everyone with psychological issues away from a free online web app. >ChatGPT should be nowhere near anyone dealing with psychological issues. Should every ledge with a >10ft drop have a suicide net? How would you imagine this would be enforced, requiring everyone who uses ChatGPT to agree to an "I am mentally stable" provisio?
- fireflash38 11mo ago
  Do you think that it's free and available to anyone means it doesn't have any responsibility to users? Or have any responsibility for how it's used, or what it says?
  - atleastoptimal 11mo ago
    It’s an open problem in AI development to make sure LLM’s never say the “wrong” thing. No matter what, when dealing with a non-deterministic system, one can’t anticipate or oversee the moral shape of all its outputs. There are a lot of things however that you can’t get ChatGPT to say, and they often ban users after successive violations, so it isn’t true that they are fully abdicating responsibility for the use and outputs of their models in realms where the harm is tractable.
Lerc 11mo ago
While I agree that AI shouldn't be sycophantic, and I also agree that the AI shouldn't have said those things. > “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.” That is not sycophantic behaviour, it is asserting a form of control of the situation. The bot made a direct challenge to the suggestion. I only just realised this now reading your comment, but I hardly ever see responses that push back against what I say like that.
prasadjoglekar 11mo ago
This is just one example of the logical end state of grossly over prioritizing capital over labor in the economy.
moralestapia 11mo ago
That's really really bad. But also, how many people has it talked out of doing it? We need the full picture.
- JKCalhoun 11mo ago
  That is data that is impossible to get.
  - moralestapia 11mo ago
    >OpenAI says over a million people talk to ChatGPT about suicide weekly
    - JKCalhoun 11mo ago
      We don't know how many of those people would have gone through with a suicide but for LLMs.
      - abracadaniel 11mo ago
        Or how many were pushed down the path towards discussing suicide because they were talking to an LLM that directed them that way. It's entirely possible the LLMs are reinforcing bleak feelings with its constant "you're absolutely correct!" garbage.
sharts 11mo ago
Something is indeed NOT better than nothing. However, for those with mental and emotional issues (likely stemming from social / societal failures in the first place) anything would be better nothing because they need interaction and patience —two things these AI tools have in abundance. Sadly there is no alternative. This is happening and there’s no going back. Many will be affected in detrimental ways (if not worse). We all go on with our lives because that which does not directly affect us is not our problem —is someone else’s problem/responsibility.
gwd 11mo ago
I feel like this article is apropos: https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced-psychosis-a-shallow-investigation https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced... Basically, the author tried to simulate someone going off into some sort of psychosis with a bunch of different models; and got wildly different results. Hard to summarize, very interesting read.
seunosewa 11mo ago
That was before they started acting to fix the problem. Please check the date.
bg24 11mo ago
There are one-off things, and then there is exponential improvements - both in guardrails, and ChatGPT's ability to handle these discussions. This type of discussion might be very much possible in ChatGPT in 6-24 months.
Xemplolo 11mo ago
This discovery was probably not on purpose. No one at OpenAI thought "Hey lets make a sucidie bot" But this should show us how shit our society is to a lot of people, how much we need to help each other. And i'm pretty sure that a good bot could def help
TZubiri 11mo ago
I know that minors under age 13 are not allowed to use the app. But 13-18 is fine? Not sure why. Might also be worth looking into making apps like these 18+. Whether by law or by liability, if someone 20+ gets, say, food poisoning by getting recipes from chatgpt, then you can argue that it's the user's fault for not fact checking, but if a 15yo kid gets food poisoning, it's harder to argue that it's the kid's fault.
- subscribed 11mo ago
  Meanwhile tide pods are still available to purchase without the age verification.

FloorEgg 11mo ago

What version of chatGPT was this? That sounds like a gpt-3.5 kind of response. I wonder how the latest models would fare in these kinds of conversations.

Al-Khwarizmi 11mo ago

We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa (not a social scientist so I have no idea what the methodology would look like, but if should be doable... or if it currently isn't, we should find the way).

computably 11mo ago
> Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa That is the literal opposite of how medical treatment is regulated. Treatments should be tested and studied before availability to the general public. It's irresponsible in the extreme to suggest this.
grey-area 11mo ago
A word generator with no intelligence or understanding based on the contents of the internet should not be allowed near suicidal teens, nor should it attempt to offer advice of any kind. This is basic common sense. Add in the commercial incentives of 'Open'AI to promote usage for anything and everything and you have a toxic mix.
- butlike 11mo ago
  Yet, if you ask the word generator to generate words in the form of advice, like any machine or code, it will do exactly what you tell it to do. The fact people are asking implies a lack of common sense by your definition. Sertraline can increase suicidal thoughts in teens. Should anti-depressants not be allowed near suicidal/depressed teens?
  - grey-area 11mo ago
    Should anti-depressants not be allowed near suicidal/depressed teens? Well certainly not without careful monitoring and medical advice, no of course not!
- Al-Khwarizmi 11mo ago
  Supposing that the advice it provides does more good than harm, why? What's the objective reason? If it can save lives, who cares if the advice is based on intelligence and understanding or on regurgitating internet content?
  - bayindirh 11mo ago
    > Supposing that the advice it provides does more good than harm, why? Because a human, esp. a confused and depressive human being is a complex thing. Much more complex than a stable, healthy human. Words encouraging a healthy person can break a depressed person further. Statistically positive words can deepen wounds, and push people more to the edge. Dark corners of human nature is twisted, hard to navigate and full of distortions. Simple words don't and can't help. Humans are not machines, brains are not mathematical formulae. We're not deterministic. We need to leave this fantasy behind.
bayindirh 11mo ago
Maybe it's causing even more deaths than we know, and these doesn't make the news either? If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news.
- Al-Khwarizmi 11mo ago
  > Maybe it's causing even more deaths than we know, and these doesn't make the news either? Of course, and that's part of why I say that we need to measure the impact. It could be net positive or negative, we won't know if we don't find out. > If we think this way, then we don't need to improve safety of anything (cars, trains, planes, ships, etc.) because we would need the big picture, though... maybe these vehicles cause death (which is awful), but it's also transporting people to their destinations alive. If there are that many people using these, I wouldn't be surprised if these actually transports some people with comfort, and that's not going to make the news. I'm not advocating for not improving security, I'm arguing against a comment that said that "ChatGPT should be nowhere near anyone dealing with psychological issues", because it can cause death. Following your analogy, cars objectively cause deaths (and not only of people with psychological issues, but of people in general) and we don't say that "they should be nowhere near a person". We improve their safety even though zero deaths is probably impossible, which we accept because they are useful. This is a big-picture approach.
jrflowers 11mo ago
[flagged]
- david-gpu 11mo ago
  They didn't say it was impossible, or that we should do nothing. Learn how to have a constructive dialogue, please.
  - jrflowers 11mo ago
    You make a good point. While they absolutely and unequivocally said that it is currently impossible to tell whether the suicides are bad or not, they also sort of wondered aloud if in the future we might be able to develop a methodology to determine whether the suicides are bad or not. This is an important distinction becau
alex-moon 11mo ago
I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. When the stakes are life or death, as they are with someone who is suicidal, that is a good example of a time when 80% isn't good enough. In such cases, where a new approach offers to replace an existing approach, the burden of proof is on the challenger, not the incumbent. This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed tests. You understand then, hopefully, why your comments here are dangerous...? I have no doubt you have no malicious intent here - you're right that these decisions need to be based on data - but you're not taking into account that the (potentially extremely harmful) challenger already has a foothold in the field. I know that you will want to hear this from experts in the "relevant field" rather than myself, so here is a write-up from Stanford on the subject: https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in...
- butlike 11mo ago
  What are you talking about? I can grow food myself, and I can build a car from scratch and take it on the highway. Are there repercussions? Sure, but nothing inherently stops me from doing it. The problem here is there's no measurable "win condition" for when a person gets good information that helps them. They remain alive, which was their previous state. This is hard to measure. Now, should people be able to google their symptoms and try and help themselves? This dovetails into a deeper philosophical discussion, but I'm not entirely convinced "seek professional help" is ALWAYS the answer. ALWAYS and NEVER are _very_ long timeframes, and we should be careful when using them.
- RobRivera 11mo ago
  I would take it a step back and posit simply, "why does a human require certification to practice therapy, and a computer program does not?" There should be some liability for malpractrice even if generated by an llm
- 11mo ago
toofy 11mo ago
people are overcomplicating this, the big picture is simple af: if a therapist was ever found to have said to this to a suicidal person, they would be immediately stripped of their license and maybe jailed.
- cal85 11mo ago
  True. But it feels like a fairer comparison would be with a huge healthcare company that failed to vet one of its therapists properly, so a crazy pro-suicide therapist slipped through the net. Would we petition to shut down the whole company for this rare event? I suppose it would depend on whether the company could demonstrate what it is doing to ensure it doesn’t happen again.
  - wat10000 11mo ago
    Maybe you shouldn't shut down OpenAI over this. But each instance of a particular ChatGPT model is the same as all the others. This is like a company that has a magical superhuman therapist that can see a million patients a day. If they're found to be encouraging suicide, then they need to be stopped from providing therapy. The fact that this is the company's only source of revenue might mean that the company has to shut down over this, but that's just a consequence of putting all your eggs in one basket.
- terminalshort 11mo ago
  But you would have to be a therapist. If a suicidal person went up to a stranger and started a conversation, there would be no consequences. That's more analogous to ChatGPT.
ml-anon 11mo ago
This entire comment section is full of wide eyed nonsense like this. It’s honestly frightening that we are even humoring this point of view.
- nihzm 11mo ago
  Since as you say this utilitarian view is rather common, perhaps it would good to show _why_ this is problematic by presenting a counterargument. The basic premise under GP's statements is that although not perfect, we should use the technology in such a way that it maximizes the well being of the largest number of people, even if comes at the expense of a few. But therein lies a problem: we cannot really measure well being (or utility). This becomes obvious if you look at individuals instead of the aggregate: imagine LLM therapy becomes widespread and a famous high profile person and your (not famous) daughter end up in "the few" for which LLM therapy goes terribly wrong and commit suicide. The loss of the famous person will cause thousands (perhaps millions) people to be a bit sad, and the loss of your daughter will cause you unimaginable pain. Which one is greater? Can they even be be compared? And how many people with a successful LLM therapy are enough to compensate for either one? Unmeasurable well-being then makes these moral calculations at best inexact and at worst completely meaningless. And if they are truly meaningless, how can they inform your LLM therapy policy decisions? Suppose for the sake of the argument we accept the above, and there is a way to measure well being. Then would it be just? Justice is a fuzzy concept, but imagine we reverse the example above: many people lose their lives because of bad LLM therapy, but one very famous person in the entertainment industry is saved by LLM therapy. Let's suppose then that this famous persons' well being plus the millions of spectators' improved well-being (through their entertainment) is worth enough to compensate the people who died. This means saving a famous funny person justifies the death of many. This does not feel just, does it? There is a vast amount of literature on this topic (criticisms of utilitarianism).
  - ml-anon 11mo ago
    This is either incredible satire or you’re a lunatic.
    - 11mo ago
computerthings 11mo ago
[dead]
hitarpetar 11mo ago
> someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa we already have an approval process for medical interventions. are you suggesting the government shut ChatGPT down until the FDA can investigate it's use for therapy? because if so I can get behind that
wat10000 11mo ago
Our standard approach for new medical treatments is to require proof of safety and efficacy before it's made available to the general public. This is because it's very, very easy for promising-looking treatments to end up being harmful. "Before declaring that it shouldn't be near anyone with psychological issues" is backwards. Before providing it to people with psychological issues, someone should study whether the positive impact is greater than the negative. Trouble is, this is such a generalized tool that it's very hard to do that.
wfleming 11mo ago
If a therapist helped 99/100 patients but tacitly encouraged the 100th to commit suicide* they would still lose their license. * ignoring the case of ethical assisted suicide for reasons of terminal illness and such, which doesn’t seem relevant to the case discussed here.
FuckButtons 11mo ago
Let’s maybe not give the benefit of the doubt to the startup which has shown itself to have the moral scruples of vault-tec just because what they’re doing might work out fine for some of the people they’re experimenting on.

bloqs 11mo ago

im willing to bet that it reduces them at a statistical level. knee jerk emotional reaction to a hallucinaton isnt the way forward with these things

torben-friis 11mo ago

>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Is that a debate worth having though? If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures. Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.

probably_wrong 11mo ago
> Is that a debate worth having though? Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions. As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop." [1] https://www.bbc.com/news/articles/cy5prvgw0r1o https://www.bbc.com/news/articles/cy5prvgw0r1o
- arowthway 11mo ago
  Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
  - latexr 11mo ago
    > Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good. But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders. https://archive.ph/2025.08.08-145022/https://www.404media.co/guy-gives-himself-19th-century-psychiatric-illness-after-consulting-with-chatgpt/ https://archive.ph/2025.08.08-145022/https://www.404media.co... > From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy. From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.

p1dda 11mo ago

Exactly, any company that offers chatbots to the public should do what Google did regarding suicide searches, remove harmful websites and provide info how to contact mental health professionals. Anything else would be corporate suicide (pun not intended).

b3lvedere 11mo ago

"One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing." And how would a layman know the difference? If i desperately need help with mental item x and i have no clue how to get help, am very very ashamed for even mentioning to ask for help about mental item x or there are actually no resources available, i will turn to anything else than nothing. Because item x still exists and is making me suffer 24/7. At least the bot pretends to listen, some humans cannot even do that.

probably_wrong 11mo ago
I think you're being too generous to the idea that it could help without any evidence. If we assume that there's therapeutic value to bringing your problems out then a diary is a better tool. And if we believe that it's the feedback what's helping, well, we have cases of ChatGPT encouraging people's psychosis. We know that a layman often doesn't know the difference between what's helpful and what isn't - that's why loving relatives end up often enabling people's addictions thinking they're helping. But I'd argue that a system that confidently gives mediocre feedback at best and actively psychotic at worst is not a system that should be encouraged simply because it's cheap. I also wanted to snarkily write "even a dog would be better", but the more I thought about it the more I realized that yes, a dog would probably be a solid alternative.
- b3lvedere 11mo ago
  I understand your explanation. But it might be quite tough for someone in a certain mental state to make rational decision, especially when they get the feeling there will never be help. Plenty of examples where people try and stay sane by talking to inert objects in desperation.

atleastoptimal 11mo ago

OpenAI tried to get rid of the excessively sycophantic model (4o) but there was a massive backlash. They eventually relented and kept it as a model offering in ChatGPT. OpenAI certainly has made mistakes with its rollouts in the past, but it is effectively impossible to keep everyone with psychological issues away from a free online web app. >ChatGPT should be nowhere near anyone dealing with psychological issues. Should every ledge with a >10ft drop have a suicide net? How would you imagine this would be enforced, requiring everyone who uses ChatGPT to agree to an "I am mentally stable" provisio?

fireflash38 11mo ago
Do you think that it's free and available to anyone means it doesn't have any responsibility to users? Or have any responsibility for how it's used, or what it says?
- atleastoptimal 11mo ago
  It’s an open problem in AI development to make sure LLM’s never say the “wrong” thing. No matter what, when dealing with a non-deterministic system, one can’t anticipate or oversee the moral shape of all its outputs. There are a lot of things however that you can’t get ChatGPT to say, and they often ban users after successive violations, so it isn’t true that they are fully abdicating responsibility for the use and outputs of their models in realms where the harm is tractable.

Lerc 11mo ago

While I agree that AI shouldn't be sycophantic, and I also agree that the AI shouldn't have said those things. > “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.” That is not sycophantic behaviour, it is asserting a form of control of the situation. The bot made a direct challenge to the suggestion. I only just realised this now reading your comment, but I hardly ever see responses that push back against what I say like that.

prasadjoglekar 11mo ago

This is just one example of the logical end state of grossly over prioritizing capital over labor in the economy.

moralestapia 11mo ago

That's really really bad. But also, how many people has it talked out of doing it? We need the full picture.

JKCalhoun 11mo ago
That is data that is impossible to get.
- moralestapia 11mo ago
  >OpenAI says over a million people talk to ChatGPT about suicide weekly
  - JKCalhoun 11mo ago
    We don't know how many of those people would have gone through with a suicide but for LLMs.
    - abracadaniel 11mo ago
      Or how many were pushed down the path towards discussing suicide because they were talking to an LLM that directed them that way. It's entirely possible the LLMs are reinforcing bleak feelings with its constant "you're absolutely correct!" garbage.

sharts 11mo ago

Something is indeed NOT better than nothing. However, for those with mental and emotional issues (likely stemming from social / societal failures in the first place) anything would be better nothing because they need interaction and patience —two things these AI tools have in abundance. Sadly there is no alternative. This is happening and there’s no going back. Many will be affected in detrimental ways (if not worse). We all go on with our lives because that which does not directly affect us is not our problem —is someone else’s problem/responsibility.

gwd 11mo ago

I feel like this article is apropos: https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced-psychosis-a-shallow-investigation https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced... Basically, the author tried to simulate someone going off into some sort of psychosis with a bunch of different models; and got wildly different results. Hard to summarize, very interesting read.

seunosewa 11mo ago

That was before they started acting to fix the problem. Please check the date.

bg24 11mo ago

There are one-off things, and then there is exponential improvements - both in guardrails, and ChatGPT's ability to handle these discussions. This type of discussion might be very much possible in ChatGPT in 6-24 months.

Xemplolo 11mo ago

This discovery was probably not on purpose. No one at OpenAI thought "Hey lets make a sucidie bot" But this should show us how shit our society is to a lot of people, how much we need to help each other. And i'm pretty sure that a good bot could def help

TZubiri 11mo ago

I know that minors under age 13 are not allowed to use the app. But 13-18 is fine? Not sure why. Might also be worth looking into making apps like these 18+. Whether by law or by liability, if someone 20+ gets, say, food poisoning by getting recipes from chatgpt, then you can argue that it's the user's fault for not fact checking, but if a 15yo kid gets food poisoning, it's harder to argue that it's the kid's fault.

subscribed 11mo ago
Meanwhile tide pods are still available to purchase without the age verification.