4 ms·

I kind of agree with you as I wouldn't use LLMs for that. But also, one cannot speak for everybody, if it's useful for someone on that context, why's that an i

I kind of agree with you as I wouldn't use LLMs for that. But also, one cannot speak for everybody, if it's useful for someone on that context, why's that an issue?

chowells 1y ago
The issue is that people in general are very easy to fool into believing something harmful is helping them. If it was actually useful, it's not an issue. But just because someone believes it's useful doesn't mean it actually is.
saubeidl 1y ago
Because it's probably not great for one's mental health to pretend a statistical model is ones friend?
lukan 1y ago
Well, because in a worst case scenario, if the pilot of that big airliner decides to do ChatGPT therapy instead of a real one and then suicides while flying, also other people feel the consequences.
- renewiltord 1y ago
  That's the worst case scenario? I can always construct worse ones. Suppose Donald Trump goes to a bad therapist and then decides to launch nukes at Russia. Damn, this therapy profession needs to be hard regulated. It could lead to the extinction of mankind.
  - andy99 1y ago
    Doc: The encounter could create a time paradox, the result of which could cause a chain reaction that would unravel the very fabric of the spacetime continuum and destroy the entire universe! Granted, that's a worst-case scenario. The destruction might in fact be very localised, limited to merely our own galaxy. Marty: Well, that's a relief.
    - anonymars 1y ago
      Good thing Biff Tanner becoming president was a silly fictional alternate reality. Phew.
- anonymars 1y ago
  Pilots don't go to real therapy, because real pilots don't get sad https://www.nytimes.com/2025/03/18/magazine/airline-pilot-mental-health.html https://www.nytimes.com/2025/03/18/magazine/airline-pilot-me...
  - oceanplexian 1y ago
    Yeah I was going to say, as a pilot there is no such thing as "therapy" for pilots. You would permanently lose your medical if you even mentioned the word to your doctor.
    - lukan 1y ago
      Not everywhere https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 "The crash was deliberately caused by the first officer, Andreas Lubitz, who had previously been treated for suicidal tendencies and declared unfit to work by his doctor. Lubitz kept this information from his employer and instead reported for duty. "
  - moralestapia 1y ago
    Fascinating read. Thanks.
    - nickthegreek 1y ago
      If this type of thing really interests you and you want to go on a wild ride, check out season 2 of nathan fielders's The Rehearsal. You dont need to watch s1.
csours 1y ago
Speaking for myself: the human mind does not seek truth or goodness, it primarily seeks satisfaction. That satisfaction happens in a context, and ever context is at least a little bit different. The scary part: It is very easy for LLMs to pick up someone's satisfaction context and feed it back to them. That can distort the original satisfaction context, and it may provide improper satisfaction (if a human did this, it might be called "joining a cult" or "emotional abuse" or "co-dependence"). You may also hear this expressed as "wire-heading"
- pmarreck 1y ago
  If treating an LLM as a bestie is allowing yourself to be "wire-headed"... Can gaming be "wire-heading"? Does the severity or excess matter? Is "a little" OK? This also reminds me of one of Michael Crichton's earliest works (and a fantastic one IMHO), The Terminal Man https://en.wikipedia.org/wiki/The_Terminal_Man https://en.wikipedia.org/wiki/The_Terminal_Man https://1lib.sk/book/1743198/d790fa/the-terminal-man.html https://1lib.sk/book/1743198/d790fa/the-terminal-man.html
oh_my_goodness 1y ago
Fuck.
TimTheTinker 1y ago
Because more than any other phenomenon, LLMs are capable of bypassing natural human trust barriers. We ought to treat their output with significant detachment and objectivity, especially when they give personal advice or offer support. But especially for non-technical users, LLMs leap over the uncanny valley and create conversational attachment with their users. The conversational capabilities of these models directly engages people's relational wiring and easily fools many people into believing: (a) the thing on the other end of the chat is thinking/reasoning and is personally invested in the process (not merely autoregressive stochastic content generation / vector path following) (b) its opinions, thoughts, recommendations, and relational signals are the result of that reasoning, some level of personal investment, and a resulting mental state it has with regard to me, and thus (c) what it says is personally meaningful on a far higher level than the output of other types of compute (search engines, constraint solving, etc.) I'm sure any of us can mentally enumerate a lot of the resulting negative effects. Like social media, there's a temptation to replace important relational parts of life with engaging an LLM, as it always responds immediately with something that feels at least somewhat meaningful. But in my opinion the worst effect is that there's a temptation to turn to LLMs first when life trouble comes, instead of to family/friends/God/etc. I don't mean for help understanding a cancer diagnosis (no problem with that), but for support, understanding, reassurance, personal advice, and hope. In the very worst cases, people have been treating an LLM as a spiritual entity -- not unlike the ancient Oracle of Delphi -- and getting sucked deeply into some kind of spiritual engagement with it, and causing destruction to their real relationships as a result. A parallel problem is that just like people who know they're taking a placebo pill, even people who are aware of the completely impersonal underpinnings of LLMs can adopt a functional belief in some of the above (a)-(c), even if they really know better. That's the power of verbal conversation, and in my opinion, LLM vendors ought to respect that power far more than they have.
- varispeed 1y ago
  [flagged]
  - TimTheTinker 1y ago
    > I've seen many therapists and [...] their capabilities were much worse I don't doubt it. The steps to mental and personal wholeness can be surprisingly concrete and formulaic for most life issues - stop believing these lies & doing these types of things, start believing these truths & doing these other types of things, etc. But were you tempted to stick to an LLM instead of finding a better therapist or engaging with a friend? In my opinion, assuming the therapist or friend is competent, the relationship itself is the most valuable aspect of therapy. That relational context helps you honestly face where you really are now--never trust an LLM to do that--and learn and grow much more, especially if you're lacking meaningful, honest relationships elsewhere in your life. (And many people who already have healthy relationships can skip the therapy, read books/engage an LLM, and talk openly with their friends about how they're doing.) Healthy relationships with other people are irreplaceable with regard to mental and personal wholeness. > I think you just don't like that LLM can replace therapist and offer better advice What I don't like is the potential loss of real relationship and the temptation to trust LLMs more than you should. Maybe that's not happening for you -- in that case, great. But don't forget LLMs have zero skin in the game, no emotions, and nothing to lose if they're wrong. > Hate to break it to you, but "God" are just voices in your head. Never heard that one before :) /s
zdragnar 1y ago
Whether the Hippocratic oath, the rules of the APA or any other organization, most all share "do no harm" as a core tenant. LLMs cannot conform to that rule because they cannot distinguish between good advice and enabling bad behavior.
- SoftTalker 1y ago
  Having an LLM as a friend or therapist would be like having a sociopath for those things -- not that an LLM is necessarily evil or antisocial, but they certainly meet the "lacks a sense of moral responsibility or social conscience" part of the definition.
- dcrazy 1y ago
  The counter argument is that’s just a training problem, and IMO it’s a fair point. Neural nets are used as classifiers all the time; it’s reasonable that sufficient training data could produce a model that follows the professional standards of care in any situation you hand it. The real problem is that we can’t tell when or if we’ve reached that point. The risk of a malpractice suit influences how human doctors act. You can’t sue an LLM. It has no fear of losing its license.
  - macintux 1y ago
    An LLM would, surely, have to: * Know whether its answers are objectively beneficial or harmful * Know whether its answers are subjectively beneficial or harmful in the context of the current state of a person it cannot see, cannot hear, cannot understand. * Know whether the user's questions, over time, trend in the right direction for that person. That seems awfully optimistic, unless I'm misunderstanding the point, which is entirely possible.
    - dcrazy 1y ago
      It is definitely optimistic, but I was steelmanning the optimist’s argument.
  - meroes 1y ago

I kind of agree with you as I wouldn't use LLMs for that. But also, one cannot speak for everybody, if it's useful for someone on that context, why's that an issue?

chowells 1y ago

The issue is that people in general are very easy to fool into believing something harmful is helping them. If it was actually useful, it's not an issue. But just because someone believes it's useful doesn't mean it actually is.

saubeidl 1y ago

Because it's probably not great for one's mental health to pretend a statistical model is ones friend?

lukan 1y ago

Well, because in a worst case scenario, if the pilot of that big airliner decides to do ChatGPT therapy instead of a real one and then suicides while flying, also other people feel the consequences.

renewiltord 1y ago
That's the worst case scenario? I can always construct worse ones. Suppose Donald Trump goes to a bad therapist and then decides to launch nukes at Russia. Damn, this therapy profession needs to be hard regulated. It could lead to the extinction of mankind.
- andy99 1y ago
  Doc: The encounter could create a time paradox, the result of which could cause a chain reaction that would unravel the very fabric of the spacetime continuum and destroy the entire universe! Granted, that's a worst-case scenario. The destruction might in fact be very localised, limited to merely our own galaxy. Marty: Well, that's a relief.
  - anonymars 1y ago
    Good thing Biff Tanner becoming president was a silly fictional alternate reality. Phew.
anonymars 1y ago
Pilots don't go to real therapy, because real pilots don't get sad https://www.nytimes.com/2025/03/18/magazine/airline-pilot-mental-health.html https://www.nytimes.com/2025/03/18/magazine/airline-pilot-me...
- oceanplexian 1y ago
  Yeah I was going to say, as a pilot there is no such thing as "therapy" for pilots. You would permanently lose your medical if you even mentioned the word to your doctor.
  - lukan 1y ago
    Not everywhere https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 "The crash was deliberately caused by the first officer, Andreas Lubitz, who had previously been treated for suicidal tendencies and declared unfit to work by his doctor. Lubitz kept this information from his employer and instead reported for duty. "
- moralestapia 1y ago
  Fascinating read. Thanks.
  - nickthegreek 1y ago
    If this type of thing really interests you and you want to go on a wild ride, check out season 2 of nathan fielders's The Rehearsal. You dont need to watch s1.

csours 1y ago

Speaking for myself: the human mind does not seek truth or goodness, it primarily seeks satisfaction. That satisfaction happens in a context, and ever context is at least a little bit different. The scary part: It is very easy for LLMs to pick up someone's satisfaction context and feed it back to them. That can distort the original satisfaction context, and it may provide improper satisfaction (if a human did this, it might be called "joining a cult" or "emotional abuse" or "co-dependence"). You may also hear this expressed as "wire-heading"

pmarreck 1y ago
If treating an LLM as a bestie is allowing yourself to be "wire-headed"... Can gaming be "wire-heading"? Does the severity or excess matter? Is "a little" OK? This also reminds me of one of Michael Crichton's earliest works (and a fantastic one IMHO), The Terminal Man https://en.wikipedia.org/wiki/The_Terminal_Man https://en.wikipedia.org/wiki/The_Terminal_Man https://1lib.sk/book/1743198/d790fa/the-terminal-man.html https://1lib.sk/book/1743198/d790fa/the-terminal-man.html

oh_my_goodness 1y ago

Fuck.

TimTheTinker 1y ago

Because more than any other phenomenon, LLMs are capable of bypassing natural human trust barriers. We ought to treat their output with significant detachment and objectivity, especially when they give personal advice or offer support. But especially for non-technical users, LLMs leap over the uncanny valley and create conversational attachment with their users. The conversational capabilities of these models directly engages people's relational wiring and easily fools many people into believing: (a) the thing on the other end of the chat is thinking/reasoning and is personally invested in the process (not merely autoregressive stochastic content generation / vector path following) (b) its opinions, thoughts, recommendations, and relational signals are the result of that reasoning, some level of personal investment, and a resulting mental state it has with regard to me, and thus (c) what it says is personally meaningful on a far higher level than the output of other types of compute (search engines, constraint solving, etc.) I'm sure any of us can mentally enumerate a lot of the resulting negative effects. Like social media, there's a temptation to replace important relational parts of life with engaging an LLM, as it always responds immediately with something that feels at least somewhat meaningful. But in my opinion the worst effect is that there's a temptation to turn to LLMs first when life trouble comes, instead of to family/friends/God/etc. I don't mean for help understanding a cancer diagnosis (no problem with that), but for support, understanding, reassurance, personal advice, and hope. In the very worst cases, people have been treating an LLM as a spiritual entity -- not unlike the ancient Oracle of Delphi -- and getting sucked deeply into some kind of spiritual engagement with it, and causing destruction to their real relationships as a result. A parallel problem is that just like people who know they're taking a placebo pill, even people who are aware of the completely impersonal underpinnings of LLMs can adopt a functional belief in some of the above (a)-(c), even if they really know better. That's the power of verbal conversation, and in my opinion, LLM vendors ought to respect that power far more than they have.

varispeed 1y ago
[flagged]
- TimTheTinker 1y ago
  > I've seen many therapists and [...] their capabilities were much worse I don't doubt it. The steps to mental and personal wholeness can be surprisingly concrete and formulaic for most life issues - stop believing these lies & doing these types of things, start believing these truths & doing these other types of things, etc. But were you tempted to stick to an LLM instead of finding a better therapist or engaging with a friend? In my opinion, assuming the therapist or friend is competent, the relationship itself is the most valuable aspect of therapy. That relational context helps you honestly face where you really are now--never trust an LLM to do that--and learn and grow much more, especially if you're lacking meaningful, honest relationships elsewhere in your life. (And many people who already have healthy relationships can skip the therapy, read books/engage an LLM, and talk openly with their friends about how they're doing.) Healthy relationships with other people are irreplaceable with regard to mental and personal wholeness. > I think you just don't like that LLM can replace therapist and offer better advice What I don't like is the potential loss of real relationship and the temptation to trust LLMs more than you should. Maybe that's not happening for you -- in that case, great. But don't forget LLMs have zero skin in the game, no emotions, and nothing to lose if they're wrong. > Hate to break it to you, but "God" are just voices in your head. Never heard that one before :) /s

zdragnar 1y ago

Whether the Hippocratic oath, the rules of the APA or any other organization, most all share "do no harm" as a core tenant. LLMs cannot conform to that rule because they cannot distinguish between good advice and enabling bad behavior.

SoftTalker 1y ago
Having an LLM as a friend or therapist would be like having a sociopath for those things -- not that an LLM is necessarily evil or antisocial, but they certainly meet the "lacks a sense of moral responsibility or social conscience" part of the definition.
dcrazy 1y ago
The counter argument is that’s just a training problem, and IMO it’s a fair point. Neural nets are used as classifiers all the time; it’s reasonable that sufficient training data could produce a model that follows the professional standards of care in any situation you hand it. The real problem is that we can’t tell when or if we’ve reached that point. The risk of a malpractice suit influences how human doctors act. You can’t sue an LLM. It has no fear of losing its license.
- macintux 1y ago
  An LLM would, surely, have to: * Know whether its answers are objectively beneficial or harmful * Know whether its answers are subjectively beneficial or harmful in the context of the current state of a person it cannot see, cannot hear, cannot understand. * Know whether the user's questions, over time, trend in the right direction for that person. That seems awfully optimistic, unless I'm misunderstanding the point, which is entirely possible.
  - dcrazy 1y ago
    It is definitely optimistic, but I was steelmanning the optimist’s argument.
- meroes 1y ago