3 ms·

>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Is that a debate worth having though? If the tool is availa

>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Is that a debate worth having though? If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures. Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.

probably_wrong 1y ago
> Is that a debate worth having though? Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions. As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop." [1] https://www.bbc.com/news/articles/cy5prvgw0r1o https://www.bbc.com/news/articles/cy5prvgw0r1o
- arowthway 1y ago
  Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
  - latexr 1y ago
    > Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good. But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders. https://archive.ph/2025.08.08-145022/https://www.404media.co/guy-gives-himself-19th-century-psychiatric-illness-after-consulting-with-chatgpt/ https://archive.ph/2025.08.08-145022/https://www.404media.co... > From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy. From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.
    - deleted 1y ago
      [deleted]
  - terminalshort 1y ago
    I've been talking about my health problems to unaccountable bullshit machines my whole life and nobody ever seemed to think it was a problem. I talked to about a dozen useless bullshit machines before I found one that could diagnose me with narcolepsy. Years later out of curiosity I asked ChatGPT and it nailed the diagnosis.
- torben-friis 1y ago
  >Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". I think it’s fair to demand that they label/warn about the intended usage, but policing it is distopic. Do car manufacturers immediately call the police when the speed limit is surpassed? Should phone manufacturers stop calls when the conversation deals with illegal topics? I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised. If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse.
  - computably 1y ago
    Those analogies are too imprecise. Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails. > I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised. > If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse. We can have both. If it is possible to have effective regulation preventing an LLM provider from storing or selling users' data, nothing would change if there were a ban on chatbots providing medical advice. OpenAI already has plenty of things it prohibits in its ToS.
    - torben-friis 1y ago
      >Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails. That's cool, but it is cool because it happens on device. Analysing your conversations for certain patterns, and either cross referencing between conversations or keeping some sort of strike system, is the exact kind of feature that is a bad turn away from a dystopia. Hopefully local llm's get good enough that we can avoid this issue altogether.
danaris 1y ago
Then... Maybe the tool should not be available universally. Maybe it should not be available to anyone. If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them, and its primary purpose is simply to bring OpenAI more profit, then maybe the world is better off without it being publicly available.
- torben-friis 1y ago
  >If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them Should we stop selling kitchen knives, packs of cards or beer as well? This is not a new problem in society. >and its primary purpose is simply to bring OpenAI more profit This is true for any product, unless you mean that it has no other purpose, which is trivially contradicted by the amount of people who decide to pay for it.
  - danaris 1y ago
    There's a qualitative difference between knives and publicly-available LLMs. Knives aren't out there actively telling you "use me, slit those wrists, then it'll all be over".
    - torben-friis 1y ago
      I don’t disagree that they are clearly unhealthy for people who aren’t mentally well, I just differ on where the role of limiting access lies. I think it’s up to the legal tutor or medical professionals to check that, and providers should at most be asked to comply with state restrictions, the same way addicts can figure on a list to ban access to a casino. The alternative places openAI and others in the role of surveilling the population and deciding what’s acceptable, which IMO has been the big fuckup of social media regulation. I do think there is an argument for how LLMs expose interaction - the friendliness that mimics human interaction should be changed for something less parasocial-friendly. More interactive Wikipedia and less intimate relationship. Then again, the human-like behavior reinforces the fact that it’s faulty knowledge, and speaking in an authoritative manner might be more harmful during regular use.

probably_wrong 1y ago

> Is that a debate worth having though? Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions. As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop." [1] https://www.bbc.com/news/articles/cy5prvgw0r1o https://www.bbc.com/news/articles/cy5prvgw0r1o

arowthway 1y ago
Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
- latexr 1y ago
  > Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good. But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders. https://archive.ph/2025.08.08-145022/https://www.404media.co/guy-gives-himself-19th-century-psychiatric-illness-after-consulting-with-chatgpt/ https://archive.ph/2025.08.08-145022/https://www.404media.co... > From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy. From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.
  - deleted 1y ago
    [deleted]
- terminalshort 1y ago
  I've been talking about my health problems to unaccountable bullshit machines my whole life and nobody ever seemed to think it was a problem. I talked to about a dozen useless bullshit machines before I found one that could diagnose me with narcolepsy. Years later out of curiosity I asked ChatGPT and it nailed the diagnosis.
torben-friis 1y ago
>Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way". I think it’s fair to demand that they label/warn about the intended usage, but policing it is distopic. Do car manufacturers immediately call the police when the speed limit is surpassed? Should phone manufacturers stop calls when the conversation deals with illegal topics? I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised. If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse.
- computably 1y ago
  Those analogies are too imprecise. Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails. > I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised. > If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse. We can have both. If it is possible to have effective regulation preventing an LLM provider from storing or selling users' data, nothing would change if there were a ban on chatbots providing medical advice. OpenAI already has plenty of things it prohibits in its ToS.
  - torben-friis 1y ago
    >Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails. That's cool, but it is cool because it happens on device. Analysing your conversations for certain patterns, and either cross referencing between conversations or keeping some sort of strike system, is the exact kind of feature that is a bad turn away from a dystopia. Hopefully local llm's get good enough that we can avoid this issue altogether.

danaris 1y ago

Then... Maybe the tool should not be available universally. Maybe it should not be available to anyone. If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them, and its primary purpose is simply to bring OpenAI more profit, then maybe the world is better off without it being publicly available.

torben-friis 1y ago
>If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them Should we stop selling kitchen knives, packs of cards or beer as well? This is not a new problem in society. >and its primary purpose is simply to bring OpenAI more profit This is true for any product, unless you mean that it has no other purpose, which is trivially contradicted by the amount of people who decide to pay for it.
- danaris 1y ago
  There's a qualitative difference between knives and publicly-available LLMs. Knives aren't out there actively telling you "use me, slit those wrists, then it'll all be over".
  - torben-friis 1y ago
    I don’t disagree that they are clearly unhealthy for people who aren’t mentally well, I just differ on where the role of limiting access lies. I think it’s up to the legal tutor or medical professionals to check that, and providers should at most be asked to comply with state restrictions, the same way addicts can figure on a list to ban access to a casino. The alternative places openAI and others in the role of surveilling the population and deciding what’s acceptable, which IMO has been the big fuckup of social media regulation. I do think there is an argument for how LLMs expose interaction - the friendliness that mimics human interaction should be changed for something less parasocial-friendly. More interactive Wikipedia and less intimate relationship. Then again, the human-like behavior reinforces the fact that it’s faulty knowledge, and speaking in an authoritative manner might be more harmful during regular use.