8 ms·
Well, good, because these things make bad friends and worse therapists.
by jelder 1y ago
Well, good, because these things make bad friends and worse therapists.
- moralestapia 1y agoI kind of agree with you as I wouldn't use LLMs for that. But also, one cannot speak for everybody, if it's useful for someone on that context, why's that an issue?
- chowells 1y agoThe issue is that people in general are very easy to fool into believing something harmful is helping them. If it was actually useful, it's not an issue. But just because someone believes it's useful doesn't mean it actually is.
- saubeidl 1y agoBecause it's probably not great for one's mental health to pretend a statistical model is ones friend?
- lukan 1y agoWell, because in a worst case scenario, if the pilot of that big airliner decides to do ChatGPT therapy instead of a real one and then suicides while flying, also other people feel the consequences.
- renewiltord 1y agoThat's the worst case scenario? I can always construct worse ones. Suppose Donald Trump goes to a bad therapist and then decides to launch nukes at Russia. Damn, this therapy profession needs to be hard regulated. It could lead to the extinction of mankind.
- andy99 1y agoDoc: The encounter could create a time paradox, the result of which could cause a chain reaction that would unravel the very fabric of the spacetime continuum and destroy the entire universe! Granted, that's a worst-case scenario. The destruction might in fact be very localised, limited to merely our own galaxy. Marty: Well, that's a relief.
- anonymars 1y agoGood thing Biff Tanner becoming president was a silly fictional alternate reality. Phew.
- anonymars 1y agoPilots don't go to real therapy, because real pilots don't get sad https://www.nytimes.com/2025/03/18/magazine/airline-pilot-mental-health.html https://www.nytimes.com/2025/03/18/magazine/airline-pilot-me...
- oceanplexian 1y agoYeah I was going to say, as a pilot there is no such thing as "therapy" for pilots. You would permanently lose your medical if you even mentioned the word to your doctor.
- lukan 1y agoNot everywhere https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.m.wikipedia.org/wiki/Germanwings_Flight_9525 "The crash was deliberately caused by the first officer, Andreas Lubitz, who had previously been treated for suicidal tendencies and declared unfit to work by his doctor. Lubitz kept this information from his employer and instead reported for duty. "
- moralestapia 1y agoFascinating read. Thanks.
- nickthegreek 1y agoIf this type of thing really interests you and you want to go on a wild ride, check out season 2 of nathan fielders's The Rehearsal. You dont need to watch s1.
- csours 1y agoSpeaking for myself: the human mind does not seek truth or goodness, it primarily seeks satisfaction. That satisfaction happens in a context, and ever context is at least a little bit different. The scary part: It is very easy for LLMs to pick up someone's satisfaction context and feed it back to them. That can distort the original satisfaction context, and it may provide improper satisfaction (if a human did this, it might be called "joining a cult" or "emotional abuse" or "co-dependence"). You may also hear this expressed as "wire-heading"
- pmarreck 1y agoIf treating an LLM as a bestie is allowing yourself to be "wire-headed"... Can gaming be "wire-heading"? Does the severity or excess matter? Is "a little" OK? This also reminds me of one of Michael Crichton's earliest works (and a fantastic one IMHO), The Terminal Man https://en.wikipedia.org/wiki/The_Terminal_Man https://en.wikipedia.org/wiki/The_Terminal_Man https://1lib.sk/book/1743198/d790fa/the-terminal-man.html https://1lib.sk/book/1743198/d790fa/the-terminal-man.html
- oh_my_goodness 1y agoFuck.
- TimTheTinker 1y agoBecause more than any other phenomenon, LLMs are capable of bypassing natural human trust barriers. We ought to treat their output with significant detachment and objectivity, especially when they give personal advice or offer support. But especially for non-technical users, LLMs leap over the uncanny valley and create conversational attachment with their users. The conversational capabilities of these models directly engages people's relational wiring and easily fools many people into believing: (a) the thing on the other end of the chat is thinking/reasoning and is personally invested in the process (not merely autoregressive stochastic content generation / vector path following) (b) its opinions, thoughts, recommendations, and relational signals are the result of that reasoning, some level of personal investment, and a resulting mental state it has with regard to me, and thus (c) what it says is personally meaningful on a far higher level than the output of other types of compute (search engines, constraint solving, etc.) I'm sure any of us can mentally enumerate a lot of the resulting negative effects. Like social media, there's a temptation to replace important relational parts of life with engaging an LLM, as it always responds immediately with something that feels at least somewhat meaningful. But in my opinion the worst effect is that there's a temptation to turn to LLMs first when life trouble comes, instead of to family/friends/God/etc. I don't mean for help understanding a cancer diagnosis (no problem with that), but for support, understanding, reassurance, personal advice, and hope. In the very worst cases, people have been treating an LLM as a spiritual entity -- not unlike the ancient Oracle of Delphi -- and getting sucked deeply into some kind of spiritual engagement with it, and causing destruction to their real relationships as a result. A parallel problem is that just like people who know they're taking a placebo pill, even people who are aware of the completely impersonal underpinnings of LLMs can adopt a functional belief in some of the above (a)-(c), even if they really know better. That's the power of verbal conversation, and in my opinion, LLM vendors ought to respect that power far more than they have.
- varispeed 1y ago[flagged]
- TimTheTinker 1y ago> I've seen many therapists and [...] their capabilities were much worse I don't doubt it. The steps to mental and personal wholeness can be surprisingly concrete and formulaic for most life issues - stop believing these lies & doing these types of things, start believing these truths & doing these other types of things, etc. But were you tempted to stick to an LLM instead of finding a better therapist or engaging with a friend? In my opinion, assuming the therapist or friend is competent, the relationship itself is the most valuable aspect of therapy. That relational context helps you honestly face where you really are now--never trust an LLM to do that--and learn and grow much more, especially if you're lacking meaningful, honest relationships elsewhere in your life. (And many people who already have healthy relationships can skip the therapy, read books/engage an LLM, and talk openly with their friends about how they're doing.) Healthy relationships with other people are irreplaceable with regard to mental and personal wholeness. > I think you just don't like that LLM can replace therapist and offer better advice What I don't like is the potential loss of real relationship and the temptation to trust LLMs more than you should. Maybe that's not happening for you -- in that case, great. But don't forget LLMs have zero skin in the game, no emotions, and nothing to lose if they're wrong. > Hate to break it to you, but "God" are just voices in your head. Never heard that one before :) /s
- zdragnar 1y agoWhether the Hippocratic oath, the rules of the APA or any other organization, most all share "do no harm" as a core tenant. LLMs cannot conform to that rule because they cannot distinguish between good advice and enabling bad behavior.
- SoftTalker 1y agoHaving an LLM as a friend or therapist would be like having a sociopath for those things -- not that an LLM is necessarily evil or antisocial, but they certainly meet the "lacks a sense of moral responsibility or social conscience" part of the definition.
- dcrazy 1y agoThe counter argument is that’s just a training problem, and IMO it’s a fair point. Neural nets are used as classifiers all the time; it’s reasonable that sufficient training data could produce a model that follows the professional standards of care in any situation you hand it. The real problem is that we can’t tell when or if we’ve reached that point. The risk of a malpractice suit influences how human doctors act. You can’t sue an LLM. It has no fear of losing its license.
- macintux 1y agoAn LLM would, surely, have to: * Know whether its answers are objectively beneficial or harmful * Know whether its answers are subjectively beneficial or harmful in the context of the current state of a person it cannot see, cannot hear, cannot understand. * Know whether the user's questions, over time, trend in the right direction for that person. That seems awfully optimistic, unless I'm misunderstanding the point, which is entirely possible.
- dcrazy 1y agoIt is definitely optimistic, but I was steelmanning the optimist’s argument.
- meroes 1y ago
- dsadfjasdf 1y agoAre all humans good friends and therapists?
- saubeidl 1y agoNot all humans are good friends and therapists. All LLMS are bad friends and therapists.
- quantummagic 1y ago> all LLMS are bad friends and therapists. Is that just your gut feel? Because there has been some preliminary research that suggest it's, at the very least, an open question: https://neurosciencenews.com/ai-chatgpt-psychotherapy-28415/ https://neurosciencenews.com/ai-chatgpt-psychotherapy-28415/ https://pmc.ncbi.nlm.nih.gov/articles/PMC10987499/ https://pmc.ncbi.nlm.nih.gov/articles/PMC10987499/ https://arxiv.org/html/2409.02244v2 https://arxiv.org/html/2409.02244v2
- fwip 1y agoThe first link says that patients can't reliably tell which is the therapist and which is LLM in single messages, which yeah, that's an LLM core competency. The second is "how 2 use AI 4 therapy" which, there's at least one paper for every field like that. The last found that they were measurably worse at therapy than humans. So, yeah, I'm comfortable agreeing that all LLMs are bad therapists, and bad friends too.
- dingnuts 1y agothere's also been a spate of reports like this one recently https://www.papsychotherapy.org/blog/when-the-chatbot-becomes-the-crisis-understanding-ai-induced-psychosis https://www.papsychotherapy.org/blog/when-the-chatbot-become... which is definitely worse than not going to a therapist
- pmarreck 1y ago
- hn_throwaway_99 1y agoWhich is a bit frightening because a lot of the r/ChatGPT comments strike me as unhinged - it's like you would have thought that OpenAI murdered their puppy or something.
- jcims 1y agoThis is only going to get worse. Anyone that remembers the reaction when Sydney from Microsoft or more recently Maya from Sesame losing their respective 'personality' can easily see how product managers are going to have to start paying attention to the emotional impact of changing or shutting down models.
- nilespotter 1y agoOr they could just do it whenever they want to for whatever reason they want to. They are not responsible for the mental health of their users. Their users are responsible for that themselves.
- deleted 1y ago[deleted]
- AlecSchueler 1y agoGenerally it's poor business to give a big chunk of your users am incredibly visceral and negative emotional reaction to your product update.
- einarfd 1y agoDepends on what business OpenAI wants to be in. If they want to be in the business of selling AI to companies. Then "firing" the consumer customers that want someone to talk to, and double down models that are useful for work. Can be a wise choice.
- sacado2 1y agoUnless you want to improve your ratio of paid-to-free users and change your userbase in the process. They're pissing off free users, but pros who use the paid version might like this new version better.
- resource_waste 1y agoWell, like, thats just your opinion man. And probably close to wrong if we are looking at the sheer scale of use. There is a bit of reality denial among anti-AI people. I thought about why people don't adjust to this new reality. I know one of my friends was anti-AI and seems to continue to be because his reputation is a bit based on proving he is smart. Another because their job is at risk.
- monster_truck 1y agoThe number of comments in the thread talking about 4o as if it were their best friend the shared all their secrets with is concerning. Lotta lonely folks out there
- delfinom 1y agoWait until you see https://www.reddit.com/r/MyBoyfriendIsAI/ https://www.reddit.com/r/MyBoyfriendIsAI/ They are very upset by the gpt5 model
- pmarreck 1y agooh god, this is some real authentic dystopia right here these things are going to end up in android bots in 10 years too (honestly, I wouldn't mind a super smart, friendly bot in my old age that knew all my quirks but was always helpful... I just would not have a full-on relationship with said entity!)
- hdgvhicv 1y agoDon’t date robots. https://youtu.be/JPQJBgWwg3o https://youtu.be/JPQJBgWwg3o
- razster 1y agoThat subreddit is fascinating and yet saddening at the same time. What I read will haunt me.
- greesil 1y agoI weep for humanity. This is satire right? On the flip side I guess you could charge these users more to keep 4o around because they're definitely going to pay.
- abxyz 1y agohttps://www.nytimes.com/2025/08/08/technology/ai-chatbots-delusions-chatgpt.html https://www.nytimes.com/2025/08/08/technology/ai-chatbots-de...