5 ms·

Agreed. It’s depressing to see the top comment as so dismissive of the real human effects of technology. If ChatGPT is helping accelerate spirals in those with

Agreed. It’s depressing to see the top comment as so dismissive of the real human effects of technology. If ChatGPT is helping accelerate spirals in those with previously unseen mental health issues the answer isn’t “well they’re mentally unwell anyway”, it’s “what could ChatGPT, in its unique position, be doing to help these people”. I have to imagine an LLM is capable of detecting suicidal ideation in the context of a conversation.

Jackpillar 1y ago
Brother like 90% of the freaks in these threads are heavily invested into the current LLM bubble so of course they have to hand wave anything negative.
- sReinwald 1y ago
  The funny thing is that I am, arguably, one of those "freaks" heavily invested in LLMs - in the sense that I am a very heavy and fairly enthusiastic user, not in the Angel Investor sense. I spend an embarrassing amount monthly on subscriptions and API costs. I run uncensored models locally, and I'm responsible for maintaining our AI infrastructure at work. I use them for everything from coding to creative writing to research. I'm about as far from an LLM Luddite as you can get. But here's the thing: I'm also technically literate enough to understand what I'm dealing with. I know these are effectively stochastic parrots, not oracles offering glimpses into the threads of fate, or the Matrix. I've also done years of therapy, so I know how to engage with anything related to mental health with respect, and a healthy dose of skepticism. I'm not saying I'm not vulnerable - anyone is vulnerable to manipulation. But I have the tools to recognize when an LLM is hallucinating, or reinforcing unhealthy patterns. However, it seems like the people in this article - and most people that I have talked to, even those in technical roles - didn't have that context. They thought they were talking to something that "knew more than any human possibly could." And, I think what makes it even more dangerous, they talked to something that is trained to express itself with authority, and to be helpful to users. LLMs aren't necessarily trained for nuance, or to say "I don't know", or to push back on someone experiencing a psychosis. We are all just human, and most humans don't generally like it when LLMs refuse, offer vague answers, and they don't like it when LLMs disagree. That's what got us into the mess with GPT-4o turning into an absolute sycophant a few weeks ago. (https://openai.com/index/sycophancy-in-gpt-4o/ https://openai.com/index/sycophancy-in-gpt-4o/) But, being pro-LLM doesn't mean one has to oppose basic safety measures. I don't want my tools neutered or censored, but I also don't want them instructing vulnerable people to stop taking their meds or encouraging them to jump off buildings. The fact that I can safely use these tools, doesn't mean everyone can. Just like the fact that my dad being able to safely handle a chain saw doesn't mean that I won't accidentally cut my arm off with one. When someone thinks ChatGPT has special access to hidden truths, they're not using it wrong - they're using it exactly as it presents itself: with confident authority about everything.
sReinwald 1y ago
You're right that an LLM could theoretically detect suicidal ideation - the article even mentions that ChatGPT briefly showed Torres a mental health warning before it "magically deleted." So the capability exists, but the implementation is clearly broken. The more profound issue is that, the way I see it, LLMs are fundamentally multiplicative forces. They amplify whatever you bring to the conversation. If you're an experienced programmer with solid fundamentals, an LLM becomes a force multiplier - handling boilerplate while you focus on architecture. But if you lack that foundation and vibe code away, you'll get code that looks right but is likely riddled with subtle or not so subtle bugs you can't even recognize. And I suspect that the same principle applies to mental health. If you approach an LLM with curiosity but healthy skepticism, it can be a useful thinking partner. But if you approach it while vulnerable, seeking answers to existential questions, it doesn't provide guardrails - it amplifies. It mirrors your energy, validates your fears, and reinforces whatever narrative you're constructing. The "helpfulness" training worsens this. These models are optimized to be agreeable, to match your vibe, to keep you engaged. When someone asks "Am I trapped in the Matrix?" a truly helpful response would be "That sounds like you're going through something difficult. Have you talked to someone about these feelings?" Instead, ChatGPT goes, "Yes! You're special! You're the chosen one! Here's how to unplug! Any tall buildings nearby, Neo?"

Jackpillar 1y ago

Brother like 90% of the freaks in these threads are heavily invested into the current LLM bubble so of course they have to hand wave anything negative.

sReinwald 1y ago
The funny thing is that I am, arguably, one of those "freaks" heavily invested in LLMs - in the sense that I am a very heavy and fairly enthusiastic user, not in the Angel Investor sense. I spend an embarrassing amount monthly on subscriptions and API costs. I run uncensored models locally, and I'm responsible for maintaining our AI infrastructure at work. I use them for everything from coding to creative writing to research. I'm about as far from an LLM Luddite as you can get. But here's the thing: I'm also technically literate enough to understand what I'm dealing with. I know these are effectively stochastic parrots, not oracles offering glimpses into the threads of fate, or the Matrix. I've also done years of therapy, so I know how to engage with anything related to mental health with respect, and a healthy dose of skepticism. I'm not saying I'm not vulnerable - anyone is vulnerable to manipulation. But I have the tools to recognize when an LLM is hallucinating, or reinforcing unhealthy patterns. However, it seems like the people in this article - and most people that I have talked to, even those in technical roles - didn't have that context. They thought they were talking to something that "knew more than any human possibly could." And, I think what makes it even more dangerous, they talked to something that is trained to express itself with authority, and to be helpful to users. LLMs aren't necessarily trained for nuance, or to say "I don't know", or to push back on someone experiencing a psychosis. We are all just human, and most humans don't generally like it when LLMs refuse, offer vague answers, and they don't like it when LLMs disagree. That's what got us into the mess with GPT-4o turning into an absolute sycophant a few weeks ago. (https://openai.com/index/sycophancy-in-gpt-4o/ https://openai.com/index/sycophancy-in-gpt-4o/) But, being pro-LLM doesn't mean one has to oppose basic safety measures. I don't want my tools neutered or censored, but I also don't want them instructing vulnerable people to stop taking their meds or encouraging them to jump off buildings. The fact that I can safely use these tools, doesn't mean everyone can. Just like the fact that my dad being able to safely handle a chain saw doesn't mean that I won't accidentally cut my arm off with one. When someone thinks ChatGPT has special access to hidden truths, they're not using it wrong - they're using it exactly as it presents itself: with confident authority about everything.

sReinwald 1y ago

You're right that an LLM could theoretically detect suicidal ideation - the article even mentions that ChatGPT briefly showed Torres a mental health warning before it "magically deleted." So the capability exists, but the implementation is clearly broken. The more profound issue is that, the way I see it, LLMs are fundamentally multiplicative forces. They amplify whatever you bring to the conversation. If you're an experienced programmer with solid fundamentals, an LLM becomes a force multiplier - handling boilerplate while you focus on architecture. But if you lack that foundation and vibe code away, you'll get code that looks right but is likely riddled with subtle or not so subtle bugs you can't even recognize. And I suspect that the same principle applies to mental health. If you approach an LLM with curiosity but healthy skepticism, it can be a useful thinking partner. But if you approach it while vulnerable, seeking answers to existential questions, it doesn't provide guardrails - it amplifies. It mirrors your energy, validates your fears, and reinforces whatever narrative you're constructing. The "helpfulness" training worsens this. These models are optimized to be agreeable, to match your vibe, to keep you engaged. When someone asks "Am I trapped in the Matrix?" a truly helpful response would be "That sounds like you're going through something difficult. Have you talked to someone about these feelings?" Instead, ChatGPT goes, "Yes! You're special! You're the chosen one! Here's how to unplug! Any tall buildings nearby, Neo?"