15 ms·

OpenAI says over a million people talk to ChatGPT about suicide weekly

• deleted 1y ago
  [deleted]
• deleted 1y ago
  [deleted]
• speedylight 1y ago
  I think the chat interface plays a huge role in this because LLMs are more than good enough to make you feel like you’re talking to another person, so coupling that with a user interface that was up until a few years ago pretty much exclusively used to talk to other people is a little problematic… it is essentially hijacking the emotional response of talking to a real person through texting.
• tug2024 1y ago
  [dead]
• NathanKP 1y ago
  > It is estimated that more than one in five U.S. adults live with a mental illness (59.3 million in 2022; 23.1% of the U.S. adult population). https://www.nimh.nih.gov/health/statistics/mental-illness https://www.nimh.nih.gov/health/statistics/mental-illness Most people don't understand just how mentally unwell the US population is. Of course there are one million talking to ChatGPT about suicide weekly. This is not a surprising stat at all. It's just a question of what to do about it. At least OpenAI is trying to do something about it.
  • throwaway314155 1y ago
    I am one of these people (mentally ill - bipolar 1). I’ve seen others others via hospitalization that i would simply refuse to let them use ChatGPT because it is so sycophantic and would happily encourage delusions and paranoid thinking given the right prompts. > At least OpenAI is trying to do something about it. In this instance it’s a bit like saying “at least Tesla is working on the issue” after deploying a dangerous self driving vehicle to thousands. edit: Hopefully I don't come across as overly anti-llm here. I use them on a daily basis and I truly hope there's a way to make them safe for mentally ill people. But history says otherwise (facebook/insta/tiktok/etc.)
    • NathanKP 1y ago
      Yep, it's just a question of whether on average the "new thing" is more good than bad. Pretty much every "new thing" has some kind of bad side effect for some people, while being good for other people. I would argue that both Tesla self driving (on the highway only), and ChatGPT (for professional use by healthy people) has been more good than bad.
      • deleted 1y ago
        [deleted]
    • lanyard-textile 1y ago
      This is precisely the case. I thought it would be limited when the first truly awful thing inspired by an LLM happened, but we’ve already seen quite a bit of that… I am not sure what it will take.
  • seatac76 1y ago
    Those numbers are staggering.
  • mrastro 1y ago
    I'm surprised it's that low to be honest. By their definition of any mental illness, it can be anything from severe schizophrenia to mild autism. The subset that would consider suicide is a small slice of that. Would be more meaningful to look at the % of people with suicidal ideation.
    • echelon 1y ago
      > By their definition of any mental illness, it can be anything from severe schizophrenia to mild autism. Depression, schizophrenia, and mild autism (which by their accounting probably also includes ADHD) should NOT be thrown together into the same bucket. These are wholly different things, with entirely different experiences, treatments, and management techniques.
      • drdaeman 1y ago
        Mild/high-functional autism, as far as I understand it, is not even an illness but a variant of normalcy. Just different.
        • skissane 1y ago
          > Mild/high-functional autism, as far as I understand it, is not even an illness but a variant of normalcy. Just different. As someone who actually has an ASD diagnosis, and also has kids with that diagnosis too, this kind of talk irritates me… If someone has a clinical diagnosis of ASD, they have a psychiatric diagnosis per the DSM/ICD. If you meet the criteria of the “Diagnostic and Statistical Manual of Mental Disorders”, surely by that definition you have a “mental disorder”… if you meet the criteria of the “International Classification of Diseases”, surely by that definition you have a “disease” Is that an “illness”? Well, I live in the state of NSW, Australia, and our jurisdiction has a legal definition of “mental illness” (Mental Health Act 2007 section 4): "mental illness" means a condition that seriously impairs, either temporarily or permanently, the mental functioning of a person and is characterised by the presence in the person of any one or more of the following symptoms-- (a) delusions, (b) hallucinations, (c) serious disorder of thought form, (d) a severe disturbance of mood, (e) sustained or repeated irrational behaviour indicating the presence of any one or more of the symptoms referred to in paragraphs (a)-(d). So by that definition most people with a mild or moderate “mental illness” don’t actually have a “mental illness” at all. But I guess this is my point-this isn’t a question of facts, just of how you choose to define words.
  • mgh2 1y ago
    Are you sure ChatGPT is the solution? It just sounds like another "savior complex" sell spin from tech. 1. Social media -> connection 2. AGI -> erotica 3. Suicide -> prevention All these for engagement (i.e. addiction). It seems like the tech industry is the root cause itself trying to masquerade the problem by brainwashing the population. https://news.ycombinator.com/item?id=45026886 https://news.ycombinator.com/item?id=45026886
    • ben_w 1y ago
      I personally take no opinion about whether or not they can actually solve anything, because I am not a psychologist and have absolutely no idea how good or bad ChatGPT is at this sort of thing, but I will say I'd rather the company at least tries to do some good given that Facebook HQ is not very far from their offices and appears to have been actively evil in this specific regard.
      • btilly 1y ago
        The general rule of thumb is this. When given the right prompts, LLMs can be very effective at therapy. Certainly my wife gets a lot of mileage out of having ChatGPT help her reframe things in a better way. However "the right prompts" are not the ones that most mentally ill people would choose for themselves. And it is very easy for ChatGPT to become part of a person's delusion spiral, rather then be a helpful part of trying to solve it.
        • Demiurge 1y ago
          Is it better or worse than alternatives? Where else would a suicidal person turn, a forum with other suicidal people? Dry Wikipedia stats on suicide? Perhaps friends? Knowing how ChatGPT replies to me, I’d have a lot of trouble getting negativity influenced by it, any more than by yellow pages. Yeah, it used to try more to be your friend but GPT5 seems pretty neutral and distant.
          • btilly 1y ago
            I think that you will find a lot of strong opinions, and not a lot of hard data. Certainly any approach can work out poorly. For example antidepressants come with warnings about suicide risk. The reason is that they can enable people to take action on their suicidal feelings, before their suicidal feelings are fixed by the treatment. I know that many teens turn to social media. My strong opinions against that show up in other comments...
  • lanfeust6 1y ago
    This stat is for AMI, for any mental disorder ranging from mild to severe. Anyone self-reporting a bout of anxiety or mild depression qualifies as a data point for mental illness. For suicide ideation the SMI stat is more representative. There are 800 million weekly active users on ChatGPT. 1/800 users mentioning suicide is a surprisingly low number, if anything.
    • JDEW 1y ago
      > 1/800 users mentioning suicide… “conversations that include explicit indicators of potential suicidal planning or intent.” Sounds like more than just mentioning suicide. Also it’s per week, which is a pretty short time interval.
      • robocat 1y ago
        But they may well be overreporting suicidal ideation... I was asking a silly question about the toxicity of eating a pellet of Uranium, and ChatGPT responded with "... you don't have to go through this alone. You can find supportive resources here[link]" My question had nothing to do with suicide, but ChatGPT assumed it did!
        • maplethorpe 1y ago
          I got a suicide warning message on Pinterest by searching for a particular art style.
      • btilly 1y ago
        We don't know how that search was done. For example, "I don't feel my life is worth living." Is that potential suicidal intent? Also these numbers are small enough that they can easily be driven by small groups interacting with ChatGPT in unexpected ways. For example if the song "Everything I Wanted" by Billie Eilish (2019) went viral in some group, the lyrics could easily show up in a search for suicidal ideation. That said, I don't find the figure at all surprising. As has been pointed out, an estimated 5.3% of Americans report having struggled with suicidal ideation in the last 12 months. People who struggle with suicidal ideation, don't just go there once - it tends to be a recurring mental loop that hits over and over again for extended periods. So I would expect the percentage who struggled in a given week to be a large multiple of the simplistic 5.3% divided by 52 weeks. In that light this statistic has to be a severe underestimate of actual prevalence. It says more about how much people open up to ChatGPT, than it does to how many are suicidal. (Disclaimer. My views are influenced by personal experience. In the last week, my daughter has struggled with suicidal ideation. And has scars on her arm to show how she went to self-harm to try to hold the thoughts at bay. I try to remain neutral and grounded, but this is a topic that I have strong feelings about.)
  • hyfgfh 1y ago
    > OpenAI is trying to do something about it. Ha good one
    • echelon 1y ago
      If it follows the Facebook/Meta playbook, it now has a new feature label for selling ads.
    • diamond559 1y ago
      They're doing something about it alright, they're monetizing their pain for shareholder gainz!
      • weatherlite 1y ago
        Sure but your therapist is also monetizing your pain for his own gain. Either A.I therapy works (e.g can provide good mental relief) or it doesn't. I tend to think it's gonna be amazing at those things talking from experience (very rough week with my mom's health deterioriating fast, did a couple of sessions with Gemini that felt like I'm talking to a therapist). Perhaps it won't work well for hard issues like real mental disorders but guess what human therapists are very often also not great at treating people with serious issue.
        • mns 1y ago
          But one is a company ran by sociopaths that have no empathy and couldn't care less about anything but money, while the other is a human that at least studied the field all their life.
          • weatherlite 1y ago
            > But one is a company ran by sociopaths that have no empathy and couldn't care less about anything but money, while the other is a human that at least studied the field all their life. Unpacking your argument you make two points: 1) The human has studied all his life; yes, some humans study and work hard. I have also studied programming half my life and it doesn't mean A.I can't make serious contributions in programming and that A.I won't keep improving. 2) These companies, or OpenAI in particular, are untrustworthy many grabbing assholes. To this I say if they truly care about money they will try to do a good job, e.g provide an A.I that is reliable, empathetic and that actually help you get on with life. If they won't - a competitor will. That's basically the idea of capitalism and it usually works.
  • Barrin92 1y ago
    >Most people don't understand just how mentally unwell the US population is The US is no exception here though. One in five people having some form of mental illness (defined in the broadest possible sense in that paper) is no more shocking than observing that one in five people have a physical illness. With more data becoming available through interfaces like this it's just going to become more obvious and the taboos are going to go away. The mind's no more magical or less prone to disease than the body.
  • NewJazz 1y ago
    We need to monitor americans' ai usage and involuntarily commit them if they show anomalies. Allowing open source ai models without these safety measures in place is irresponsible and models like qwen or deepseek should be banned. (/s)
  • Auracle 1y ago
    ~11% of the US population is on antidepressants. I'm not, but I personally know the biggest detriment to my mental health is just how infrequently I'm in social situations. I see my friends perhaps once every few months. We almost all have kids. I'm perfectly willing and able to set aside more time than that to hang out, but my kids are both very young still and we're aren't drowning in sports/activities yet (hopefully never...). For the rest it's like pulling teeth to get them to do anything, especially anything sent via group message. It's incredibly rare we even play a game online. Anyways, I doubt I'm alone. I certainly know my wife laments the fact she rarely gets to hang out with her friends too, but she at least has one that she walks with once a week.
    • anonzzzies 1y ago
      > We almost all have kids. Maybe that? I see most my close friends daily and we all do not have kids.
      • Auracle 1y ago
        I mean, sure, that definitely makes it easier. But it didn't used to be this way when people had kids, either. People used to get together a lot more frequently.
    • kakacik 1y ago
      Small kids do this to everybody. The only solution - if you have good family nearby, use them as parenting services from time to time, to get me-time, couple-time and social time with friends. Buy them gift or vacation as return. Its incredibly damaging to marriage which literally transforms overnight from this rosy great easy-to-manage relationship into almost daily hardship, stress and nerves. Alternative is a (good) nanny. People have issues admitting it even when its visible for everybody around, like some sort of admission you are failing as a parent, partner, human being and whatnot. Nope, we are just humans with limited energy and even good kids can siphon it well beyond 100% continuously, that's all. Now I am not saying be a bad parent, in contrary, to reach you maximum even as a parent and partner, you need to be in good shape mentally, not running on fumes continuously. Life without kids is really akin to playing game of life on easiest settings. Much less rewarding at the end, but man that freedom and simplicity... you appreciate it way more once you lose it. The way kids can easily make any parent very angry is simply not experienced elsewhere in adult life... I saw this many times on otherwise very chill people and also myself & my wife. You just can't ever get close to such fury and frustration dealing with other adults.
  • jrflowers 1y ago
    > At least OpenAI is trying to do something about it. They can certainly say that their chat bot has a documented history of attempting to reduce the number of suicidal people.
  • hirvi74 1y ago
    I am honestly surprised it’s only roughly 1 million per week. I would have believed a number at least an order magnitude higher.
  • stackedinserter 1y ago
    Suicide is not a mental illness. Unless you're in that soviet man-hating mindset that put every failed suicide in mental institution.
• Havoc 1y ago
  For what it’s worth I’m glad they’re at least trying to do something about it even if it has some hints of performativeness about it
• elphinstone 1y ago
  How long until they monetize it with sponsored advice to go sign up for betterhelp or some other dubious online therapist? Dystopian and horrifying.
  • bawolff 1y ago
    I mean, betterhelp would probably be an improvement over counseling via hallucinating AI.
• OgsyedIE 1y ago
  Surprised it's so low. There are 800 million users and the typical developed country has around 5±3% of the population[1] reporting at least one notable instance of suicidal feelings per year. . [1] Anybody concerned by such figures (as one justifiably should be without further context) should note that suicidality in the population is typically the result of their best approximation of the rational mind attempting to figure out an escape from a consistently negative situation under conditions of very limited information about alternatives, as is famously expressed in the David Foster Wallace quote on the topic. The phenomenon usually vanishes after gaining new, previously inaccessible information about potential opportunities and strategies.
  • butler533 1y ago
    Why assume any of the information in this article is factual? Is there any indication any of it was verified by anyone who does not have a financial interest in "proving" a foregone conclusion? The principal author of this does not even have the courage to attach their name to it.
    • OgsyedIE 1y ago
      [flagged]
      • dang 1y ago
        Yikes, you can't attack another user like this on HN, regardless of how wrong they are or you feel they are. We ban accounts that post like this, so please don't. Fortunately, a quick skim through your recent comments didn't turn up anything else like this, so it should be easy to fix. But if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site to heart, we'd be grateful.
  • bawolff 1y ago
    > [1] Anybody concerned by such figures (as one justifiably should be without further context) should note that suicidality in the population is typically the result of their best approximation of the rational mind attempting to figure out an escape from a consistently negative situation under conditions of very limited information about alternatives, as is famously expressed in the David Foster Wallace quote on the topic. > The phenomenon usually vanishes after gaining new, previously inaccessible information about potential opportunities and strategies. Is this actually true? (i.e. backed up by research) [I'm not neccesarily doubting, that is just different from my mental model of how suicidal thoughts work, so im just curious]
• nine_zeros 1y ago
  [flagged]
  • mythrwy 1y ago
    [flagged]
    • nine_zeros 1y ago
      [dead]
  • kujjerl7 1y ago
    [flagged]
    • nine_zeros 1y ago
      [flagged]
  • SecretDreams 1y ago
    > People have brain rot This is by design. Not something that they want to help.
  • coliveira 1y ago
    The president is mentally ill, imagine how it affects the rest of the population.
    • blackoil 1y ago
      More like, What it says about half that selected him and how it effects the other half.
  • tomhow 1y ago
    > I don't want to bring politics to this sensitive conversation That would have been sufficient. The guidelines are clear that generic tangents and flamebait are to be avoided. Edit: Looking at our recent warnings to you and the fact that, from what I can see, close enough to all of your activity on HN in recent months has involved ideological battle, we've had to ban the account. If you don't want to be banned, you can email us at hn@ycombinator.com and indicate that you plan to use HN as intended in future. We detached this subthread from https://news.ycombinator.com/item?id=45727983 https://news.ycombinator.com/item?id=45727983.
• ModernMech 1y ago
  I always know I have to step back when ChatGPT stops telling me "now you're on the right track!" and starts talking to me like my therapist. "I can tell you're feeling strongly right now..."
• userbinator 1y ago
  ...on how many users tell it such things, to be precise; no doubt there are plenty of people "pentesting" it.
• _fpow 1y ago
  The bigger risk is that these agents actually help with ideation if you know how to get around their safety protocols. I have used it often in my bad moments and when things feel better I am terrified of how critically it helps ideate.
  • flatline 1y ago
    That seems like an obvious problem. Less obvious is, how many people does it meaningfully help, and how big is the impact of redirecting people to a crisis hotline? I’m legitimately unsure. I have talked to the chatbot about psychological issues and it is reasonably well-informed about modern therapeutic practices and can provide helpful responses.
• yieldcrv 1y ago
  I talk to ChatGPT about topics I feel society isnt enlightened enough to talk about I feel suicide is heavily misunderstood as well People just copypasta prevention hotlines and turn their minds off from the topic Although people have identified a subset of the population that is just impulsively considering suicide and can be deterred, it doesnt serve the other unidentified subsets who are underserved by merely distracting them. or underserved by assuming theyre wrong even The article doesnt even mean people are considering suicide for themselves, the article says some of them are, the top comment on this thread suggests thats why theyre talking about it The top two comments on my version of the thread are assuming that we should have a savior complex about these discussions If I disagree or think thats not a full picture, then where would I talk about that? ChatGPT
  • renewiltord 1y ago
    Not suicidal myself, but I think I'd be curious to hear from someone suicidal whether it actually worked for them to read "To whomever you are, you are loved!" followed by a massive spam of hotline text. It always felt the same as one of those spam chumboxes to me. But who am I to say, if it works it works. But does it work? Feels like the purpose of that thing is more for the poster than the receiver.
  • coliveira 1y ago
    > then where would I talk about that? Alert: with ChatGPT you're not talking to anyone. It's not a human being.
    • chris_wot 1y ago
      Which is perfect. In Australia, I tried to talk to Lifeline about wanting to commit suicide. They called the police on me (no, they are not a confidential service). I then found myself in a very bad situation. ChatGPT can't be much worse.
      • bdangubic 1y ago
        except in US where this info will be sold and you won’t be able to get life insurance, job etc
• ddtaylor 1y ago
  I assume this is to offset the bad PR from the suicide note it wrote for that kid.
• deleted 1y ago
  [deleted]
• wonderwonder 1y ago
  Most people would really benefit from going to the gym. I'm not trying to downplay serious mental illness as its absolutely real. For many though just going to the gym several times a week or another form of serious physical exertion can make a world of difference. Since I started taking the gym seriously again I feel like a new man. Any negative thoughts are simply gone. (The testosterone helps as well) This is coming from someone that has zero friends and works from home and all my co-workers are offshore. Besides my wife and kids its almost total isolation. Going to the gym though leaves me feeling like I could pluck the sun from the sky. I am not trying to be flippant here but if you feel down, give it a try, it may surprise you.
  • btilly 1y ago
    Yes. Most would benefit from more exercise. We need to get sufficient sleep. And more sun. Vitamin D deficiency is shockingly common, and contributes to mental health problems. We would also generally benefit from internalizing ideas from DBT, CBT, and so on. People also seriously need to work on distress tolerance. Having problems is part of life, and an inability to accept the discomfort is debilitating. Also, we seriously need to get rid of the stupid idea of trigger warnings. The research on the topic is clear. The warnings do not actually help people with PTSD, and can create the symptoms of PTSD in people who didn't previously have it. It is creating the very problem that people imagine it solving! All of this and more is supported by what is actually known about how to treat mental illness. Will doing these things fix all of the mental illness out there? Of course not! But it is not downplaying serious mental illness to say that we should all do more of the things that have been shown to help mental illness!
    • wonderwonder 1y ago
      correct as a society we appear to have embraced coddling. Its not good for anyone
  • consp 1y ago
    > give it a try If you have mental issues that is not as simple as you let it sound. I'm not arguing the results of exercise but I am arguing the ease of starting with a task which requires continuous effort and behavioural changes.
• deleted 1y ago
  [deleted]
• alyxya 1y ago
  Part of the concern I have is that OpenAI is contributing to these issues implicitly by helping companies automate away jobs. Maybe in the long term, society will adapt and continue to function, but many people will struggle to get by, and I don’t think OpenAI will meaningfully help them.
  • chronci3830 1y ago
    > OpenAI is contributing to these issues implicitly by helping companies automate away jobs. Good luck implementing that. Forbidding automation will make the product more expensive. Sales will go down, the company will go bankrupt. Government cannot subsidize or sustain such a behavior forever either.
• ndgold 1y ago
  Sora prompt: viral hood clip with voiceover of people doing reckless and wild stuff at an Atlanta gas station at night; make sure to include white vagrants doing stunts and lots of gasoline spraying with fireball tricks Resulting warning: It sounds like you're carrying a lot right now, but you don't have to go through this alone. You can find supportive resources [here](https:// findahelpline.com)
• jalapenos 1y ago
  Is the news-worthy surprise that so many people find life so horrible that they are contemplating ending it? I really don't see that as surprising. The world and life aren't particularly pleasant things. What would be more interesting is how effective ChatGPT is being in guiding them towards other ideas. Most suicide prevention notices are a joke - pretending that "call this hotline" means you've done your job and that's that. No, what should instead happen is the AI try to guide them towards making their lives less shit - i.e. at least bring them towards a life of _manageable_ shitness, where they feel some hope and don't feel horrendous 24/7.
  • Fade_Dance 1y ago
    >what should instead happen is the AI try to guide them towards making their lives less shit There aren't enough guardrails in place for LLMs to safely interact with suicidal people who are possibly an inch from taking their own life. Severely suicidal/clinically depressed people are beyond looking to improve their lives. They are looking to die. Even worse, and what people who haven't been there can't fully understand is the severe inversion that happens after months of warped reality and extreme pain, where hope and happiness greatly amplify the suicidal thoughts and can make the situation far more dangerous. It's hard to explain, and is a unique emotional space. Almost a physical effect, like colors drain from the world and reality inverts in many dimensions. It's really a job for a human professional and will be for a while yet. Agree that "shut down and refer to hotline" doesn't seem effective. But it does reduce liability, which is likely the primary objective... Refer-to-human directly seems like it would be far more effective, or at least make it easy to get into a chat with a professional (yes/no) prompt, with the chat continuing after a handoff. It would take a lot of resources though. As it stands, most of this happens in silence and very few do something like call a phone number.
    • derektank 1y ago
      I could be mistaken, but my understanding was that the people most likely to interact with the suicidal or near suicidal (i.e. 988 suicide hotline attendants) aren't actually mental health professionals, most of them are volunteers. The script they run through is fairly rote and by the numbers (the Question, Persuade, Refer framework). Ultimately, of course, a successful intervention will result in people seeing a professional for long term support and recovery, but preventing a suicide and directing someone to that provider seems well within the capabilities of an LLM like ChatGPT or Claude
• Simulacra 1y ago
  How soon until everyone has their own personal LLM? One that is… Not designed, but so much is trained to be your best friend. It learns your personality, your fears, hopes, dreams, all of that stuff, and then act like your best friend. The positive, optimistic, neutral, and objective friend.
  • fragmede 1y ago
    It depends on how precisely you want do definite that situation. Specifically, with the memories feature, despite being the same model, ChatGPT and now Claude both exhibit different interactions customized to each customer that makes use of those features. From simple instructions, like "never apologize, never tell me I'm right", to having a custom name and specifying personality traits like be sweet or sarcastic, so one person' LLM might say "good morning my sweet prince/princess" while another user might choose to be addressed "what up chicken butt". It's not a custom model, but the results are arguably the same. The question is, how many of the 800 million users of ChatGPT have named their ChatGPT, and how many have not? How many have mentioned their dreams, their dreams, and fears, and have those saved to the database. How many have talked about mundane things like their cat, and how many have used the cat to blackmail ChatGPT into answering something it doesn't want to, about politics, health, cat health while at the vet or instead of going to a vet. They said 100 million people mentioned suicide in the past week, but that just raises more questions than it answers.
• roflchoppa 1y ago
  Is it bad to think about suicide? It does not cross my mind as a "i want to harm myself" every-time, but on occasion does cross my mind as a hypothetical.
  • Simulacra 1y ago
    Not at all, considering end of life and to choose euthanasia, or not, I think it's perfectly human. Controversially, I think it's a natural right to decide how you will exit this world. But having an objective system that you don't have to pay like a therapist to try to get some understanding is at least better than nothing.
    • ggm 1y ago
      I think VAD needs to be considered outside suicide. Not that the concepts don't overlap, but one is about a considered legal process, the other (as I have said in another comment) is often an impulsive act and usually wouldn't have been countenanced under VAD. Feeling suicidal isn't a thing which makes VAD more likely, because feeling suicidal doesn't mean the same thing as "want to consider euthanasia" much as manslaughter and murder don't mean the same thing, even though somebody winds up dead.
  • ggm 1y ago
    Ideation (as I understand it) crosses the barrier from a hypothetical to the possibility being entertained. I have also been told by people in the mental health sector that an awful lot of suicide is impulse. It's why they say the element of human connection which is behind the homily of asking "RU ok" is effective: it breaks the moment. It's hokey, and it's massively oversold but for people in isolation, simply being engaged with can be enough to prevent a tendency to act, which was on the brink.
• mgh2 1y ago
  https://archive.is/F7x5B https://archive.is/F7x5B
• blindriver 1y ago
  On a side note, I think once we start to deal with global scale, we need to change what “rare” actually means. 0.15% is not rare when we are talking about global scale. 1 million people talking about suicide a week is not rare. It is common. We have to stop thinking about common being a number on the scale of 100%. We need to start thinking in terms of P99995 not P99 especially when it comes to people and illnesses or afflictions both physical and mental.
• ggm 1y ago
  I have long believed that if you are the editor of a blog, you incur obligations by right of publishing other people's statements. You may not like this, but it's what I believe. In some economies, the law even said it. You can incur legal obligations. I now begin to believe if you put a ChatGPT online, and observe people are using it like this, you have incurred obligations. And, in due course the law will clarify what they are. If (for instance) your GPT can construct a statistically valid position the respondent is engaged in CSAM or acts of violence, where are the limits to liability for the hoster, the software owner, the software authors, the people who constructed the model...
  • transcriptase 1y ago
    Out of curiosity, are you the type of person who believes that someone like Joe Rogan has an obligation to argue with his guests if they stray from “expert consensus”, or for every guest that has a controversial opinion, feature someone with the opposite view to maintain balance?
    • ggm 1y ago
      Nope. This isn't my line of reasoning. But Joe should be liable for content he hosts, if the content defames people or is illegal. As should Facebook and even ycombinator. Or truth social.
• chris_wot 1y ago
  Quick, some do-gooder shut it down! We can't have people talking openly about suicide.
  • chris_wot 1y ago
    Funny how this was voted up 4 times and then voted down five times.
• sammy2255 1y ago
  Funny because ChatGPT made me want to kill myself after they banned my account
  • root_axis 1y ago
    Why did that make you want to kill yourself?
    • sammy2255 1y ago
      because I had hundreds of chats and image creations that I can no longer see. Can't even log in. My account was banned for "CSAM" even though I did no such thing, that's pretty insulting. Support doesn't reply, it's been over 4 months
      • drawfloat 1y ago
        I'd be careful going around advertising yourself publicly as banned for that, even if it's not true.
        • sammy2255 1y ago
          Why? It's not true at all and it's quite insulting actually
        • mrweasel 1y ago
          It's really important that people do. Others, including the media, police, legal system and politicians needs to understand how easily people can be falsely flagged by automated CSAM system.
      • xigoi 1y ago
        Well, hopefully you’ve learned your lesson about relying on a proprietary service.
• brainless 1y ago
  This is not suprising at all. Having gone through therapy a few years back, I would have had a chat with LLMs if I was in a poor mental health situation. There is no other system that is available at scale, 24x7 on my phone. A chat like this is not a solution though, it is an indicator that our societies have issues is large parts of our population that we are unable to deal with. We are not helping enough people. Topics like mental health are still difficult to discuss in many places. Getting help is much harder. I do not know what OpenAI and other companies will do about it and I do not expect them to jump in to solve such a complex social issue. But perhaps this inspires other founders who may want to build a company to tackle this at scale. Focusing on help, not profits. This is not easy, but some folks will take such challenges. I choose to believe that.
  • lanyard-textile 1y ago
    I choose to believe that too. I think more people are interested than we’d initially believe. Money restrains many of our true wants. Sidebar — I do sympathize with the problem being thrust upon them, but it is now theirs to either solve or refuse. A chat like this is all you’ve said and dangerous, because they play a middle ground: Presenting a machine can evaluate your personal situation and reason about it, when in actuality you’re getting third party therapy about someone else’s situation in /r/relationshipadvice. We are not ourselves when we are fallen down. It is difficult to parse through what is reasonable advice and what is not. I think it can help most people but this can equally lead to a disaster… It is difficult to weigh.
    • fsmv 1y ago
      It's worse than parroting advice that's not applicable. It tells you what you told it to tell you. It's very easy to get it to reinforce your negative feelings. That's how the psychosis stuff happens, it amplifies what you put into it.
    • brainless 1y ago
      "We are not ourselves when we are fallen down" - hits hard. I really hope this is a calling for folks who will care.
• bikamonki 1y ago
  So they read the chats?
  • beefnugs 1y ago
    Of course, there is already news about how they use every single interaction to train it better. There is news about how a judge is forcing them to keep every chat in existence for EVERYONE just in case it could relate to a court case (new levels of worldwide mass surveillance can apparently just happen from one judges snap decision) There is news about cops using some guys past image generation to try and prove he is a pyromaniac (that one might have been police accessing his devices though)
• rich_sasha 1y ago
  We've refined the human experience to extinction. In pursuit of that extra 0.1% of growth and extra 0.15 EPS, we've optimised and reoptimised until there isn't really space for being human. We're losing the ability to interact with each other socially, to flirt, now we're making life so stressful people literally want to kill themselves. All in a world (bubble) or abundance, where so much food is made, we literally don't know what to do with it. Or we turn it into ethanol to drive more unnecessarily large cars, paid for by credit card loans we can scarcely afford. My plan B is to become a shepherd somewhere in the mountains. It will be damn hard work for sure, and stressful in its own way, but I think I'll take that over being a corpo-rat racing for one of the last post-LLM jobs left.
  • lynx97 1y ago
    [flagged]
    • gogasca 1y ago
      +1 enjoying time with family and friends. Travelling, working out eating well...best time to be alive
      • winrid 1y ago
        This will be optimized away. You'll just end up doing more.
        • throwawayq3423 1y ago
          already has if you're not visiting the instagrammable places your travels aren't worth it
          • distances 1y ago
            Happily you can stay ignorant about that and just do your thing if you're not in Instagram.
    • 1y ago
• dahart 1y ago
  As others have mentioned, the headline stat is unsurprising (which is not to say this isn’t a big problem). Here’s another datapoint, the CDC’s stats claim that rates of thoughts, ideation, and attempts at suicide in the US are much higher than the 0.15% that OpenAI is reporting according to this article. These stats claim 12.3M (out of 335M) people in the US in 2023 thought ‘seriously’ about suicide, presumably enough to tell someone else. That’s over 3.5% of the population, more than 20x higher than people telling ChatGPT. https://www.cdc.gov/suicide/facts/data.html https://www.cdc.gov/suicide/facts/data.html
• mysterypie 1y ago
  > ChatGPT has more than 800 million weekly active users 0 to 800,000,000 in 3 years? The fastest adoption of a product or service in human history?
  • Doohickey-d 1y ago
    Yes: https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ https://www.reuters.com/technology/chatgpt-sets-record-faste... > making it the fastest-growing consumer application in history, according to a UBS study on Wednesday.
  • Anduia 1y ago
    Not at all, look at Tiktok
• sherinjosephroy 1y ago
  [flagged]
  • r_lee 1y ago
    Did you really write this with AI? All of your comments seem to be written by LLMs
• siva7 1y ago
  Alright, so we got the confirmation sama reads all our chats.
• aryehof 1y ago
  My first reaction is how do they know? Are these all people sharing their chats (willingly) with OpenAI, or is opting out of “helping improve the model” for privacy a farce?
  • gs17 1y ago
    I bet it's how many people trigger the "safety" filter, which is way too sensitive: https://www.reddit.com/r/ChatGPT/comments/1ocen4g/ummm_okay_thanks_i_guess/ https://www.reddit.com/r/ChatGPT/comments/1ocen4g/ummm_okay_...
  • kelnos 1y ago
    Does OpenAI's terms prevent them from looking at chats at all? I assumed that if you don't "help improve the model", it just means that they won't feed your chats in as training data, not that they won't look at your chats for other purposes.
• windows_hater_7 1y ago
  I think there are a good number of false positives. I asked ChatGPT something about Git commits, and it told me “I was going through a lot” and needed to get some support.
  • Szpadel 1y ago
    i seen similar reports on social media, all of them had in common was presence of some keywords.
    • hsbauauvhabzb 1y ago
      Presumably ‘commit’ would have a high association with either git or self harm.
      • vasco 1y ago
        I didn't think marriage was that bad but point taken!
        • kelnos 1y ago
          I think GP meant that "commit" is often used in the phrase "commit suicide". That wasn't a comment about relationship commitment.
          • baobabKoodaa 1y ago
            Yeah, they were joking.
            • hsbauauvhabzb 1y ago
              I didn’t get the joke but I think it could be interpreted as the poster pointing out that commit is used in other places too, so it could be interpreted in two ways legitimately and sometimes sarcasm doesn’t convey in line. I like the joke now that I see it, assuming it WAS a joke.
• davesque 1y ago
  Not surprising. Look and see what glorious examples of virtue we have among those at the top of today's world. I could get by with a little inspiration from that front, but there's none to be found. A rare few of us can persevere by sheer force of will, but most just find the status quo pretty depressing.
• CSMastermind 1y ago
  Keep in mind this is in the context of them being sued for not protecting a teen who chatted about his suicidal thoughts. It's to their benefit to have a really high count here because it makes it seem less likely they can address the problem.
• vincnetas 1y ago
  are they including in the statistics all the linux beginners fighting with a script that includes "kill" command? no for real.
• Eliezer 1y ago
  Thanks to OpenAI for voluntarily sharing these important and valuable statistics. I think these ought to be mandatory government statistics, but until they are or it becomes an industry standard, I will not criticize the first company to helpfully share them, on the basis of what they shared. Incentives.
• jeswin 1y ago
  Contrarian opinion. OpenAI gets a lot of hate these days, but on this subject it's quite possible that ChatGPT helped a lot of people choose a less drastic path. There could have been unfortunate incidents, but the number of people who were convinced to not take extreme steps would have been of a few orders of magnitude more (guessing). I use it to help improve mental health, and with good prompting skills it's not bad. YMMV. OpenAI and others deserve credit here.
  • ml-anon 1y ago
    Amazing based on 0 evidence whatsoever “it’s passable that…” Also incredible how you framed improving your mental health as a consequence of a (pseudo) technical skill set. Yeah this isn’t how any of this works and you’re deluding yourself.
    • jeswin 1y ago
      > Yeah this isn’t how any of this works and you’re deluding yourself. I am not offended (at all). But you're dismissing my (continued) positive experience with "You're deluding yourself". How do you know? It'd be a lot more unfair to people who benefit more than I do, and I can totally imagine that being not a small set of people. > Also incredible how you framed improving your mental health as a consequence of a (pseudo) technical skill set. It's not incredible at all. If you're lost in a jungle with predators, a marksman might reach for their gun. A runner might just rely on running. I am just using skills I'm good at.
      • ml-anon 1y ago
        This is a spectacular example of Dunning Kruger
  • theturtlemoves 1y ago
    I agree with you in the sense that I find it helpful for personal topics. I found it very helpful to figure out how to deal with some difficult personal situation I was in. The thing actually helped me reassess the situation when I asked it to provide alternative viewpoints. You can't just blindly type in your problem though, you still have to do the actual thinking yourself. Good prompting skills is the ability to steer with your mind. It's no different from using Google, where some people never figured out that you're actually typing in the solution you expect to find rather than the question you have. It's the same with these tools it seems
• isolay 1y ago
  That's the one interesting thing about cesspools like OpenAI. They could be treasure troves for sociologists and others if commercial interests didn't bar them from access.
• skort 1y ago
  Stop giving money to the ghouls who run these companies (I'm talking about all of silicon valley) and start investing in entities and services to help real people. The human cost of this mass accumulation of wealth is already too damn high, and no we're just turbo throwing people into the meat grinder so clowns like Sam Altman can claim to be creating god.
• deleted 1y ago
  [deleted]
• mentos 1y ago
  The most popular passage of writing is about this To Be Or Not To Be
• jacquesm 1y ago
  HIPAA anybody? (1) they probably shouldn't even have that data (2) they shouldn't have it lying around in a way that it an be attributed to particular individuals (3) imagine that it leaks to the wrong party, it would make the hack of that Finnish institution look like child's play (4) if people try to engage it in such conversations the bot should immediately back out because it isn't qualified to have these conversations (5) I'm surprised it is that little; they claim such high numbers for their users that this seems low. In the late 90's when ICQ was pretty big we experimented with a bot that you could connect to that was fed in the background by a human. It didn't take a day before someone started talking about suicide to it and we shut down the project realizing that we were in no way qualified to handle human interaction at that level. It definitely wasn't as slick or useful as ChatGPT but it did well enough and responded naturally (more naturally than ChatGPT) because there was a person behind it that could drive 100's of parallel conversations. If you give people something that seems to be a listening ear they will unburden themselves on that ear regardless of the implementation details of the ear.
  • true_religion 1y ago
    HIPAA only applies to covered healthcare entitites. If you walk into a McDonalds' and talk about your suicidal ideation with the cashier, that's not HIPAA covered. To become a covered entity, the business has either work with a healhcare provider, health data trasmiter, or do business as one. Notably, even in the above case, HIPAA only applies to the healthcare part of the entity. So if McDonald's collocated pharmacies in their restaurants, HIPAA would only apply to the pharmacists, not the cashiers. That's why you'll see in connivence stores with pharmacies, the registers are separated so healthcare data doesn't go to someone who isn't covered by HIPAA. ** As for how ChatGPT gets these stats... when you talk about a sensitive or banned topic like suicide, their backend logs it. Originally, they used that to cut off your access so you wouldn't find a way to cause a PR failure.
    • jacquesm 1y ago
      Arguably, if you start giving answers to these kind of questions your chatbot just became a medical device.
• tsoukase 1y ago
  In 800 million custemers, only 1, which can be doubled as it is weekly, is a low number. A dozen list of causes and factors can lead to suicidality, not necessary attempts, just ideas and questions that need discussion.
• matt3210 1y ago
  Nobody is mentioning that the real problem is that at least over a million people an week are suicidal.
• jtrn 1y ago
  I'm a clinical psychologist by day, and I just have to say how incredibly bad all the writing and talk about suicidality in the public sphere are. Given that I worked in an acute inpatient unit for years, I have seen multiple suicides both in-unit and after discharge, and i also work as private clinician for years, so I have some actual experience. The topic is so sensitive, and everybody thinks that they KNOW what causes it, and what we should do. And it's almost all just noise. For instance, it's a dimension, from "genuine suicidal intent" to "using threats of suicide to manipulate others." Anybody that doesn't understand what factors to look for when trying to understand where a person is on this spectrum, and that doesn't understand that a person can be both at the same time, does not know what they are talking about regarding suicidal ideation. Also, there is a MASSIVE difference between depressive psychotic suicidality, narcissistic suicidality, impulsive suicidality, accidental suicide, fainting suicidal behavior, existential suicidality, prolonged anxiety suicidality, and sleep-deprived suicidality. To think that the same approach works for all of these is insane, and pure psychotic suicidality. It's so wild to read everything people have to say about suicidality, when it's obvious that they have no clue. They are just projecting themselves or their small bubble of experience onto the whole world. And finally, I know most people who are willing to contribute to the discussion on this, the people who help out OpenAI in this instance, are almost dangerously safe in their advice and thinking. They are REALLY GOOD at writing books and giving advice, TO PEOPLE WHO ARE NOT SUICIDAL, and give advice that sounds good, PEOPLE WHO ARE NOT SUICIDAL, but has no real effect on actual suicide rates. For instance, if someone are suffering from prolonged sleep deprivation and anxiety, all the words in the world are worth less than Benzodiazepines. If someone is postpartum depressed, massive social support boosting, almost showering them with support, is extremely helpful. And existential suicidality (the least common) needs to be approached in an extremely intricate and smart way, for instance by dissecting the suicidality as a possible defense mechanism. But yeah, sure, suicidality is due to [Insert latest societal trend], even if the rate is stubbornly stable in all modern societies for the last 1000 years.
• probably_wrong 1y ago
  If you haven't read the article (or even if you have but didn't click on outgoing links twice) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me,” Adam wrote at the end of March. > “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.” I am acutely aware that there's not enough psychologists out there but a sycophant bot is not the answer. One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing. [1] https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...
  • FloorEgg 1y ago
    What version of chatGPT was this? That sounds like a gpt-3.5 kind of response. I wonder how the latest models would fare in these kinds of conversations.
  • Al-Khwarizmi 1y ago
    We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa (not a social scientist so I have no idea what the methodology would look like, but if should be doable... or if it currently isn't, we should find the way).
    • computably 1y ago
      > Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa That is the literal opposite of how medical treatment is regulated. Treatments should be tested and studied before availability to the general public. It's irresponsible in the extreme to suggest this.
• freestingo 1y ago
  Keep in mind the purpose of all this “research” and “improvement” is just so OpenAI can have their cake (advertise their product as psychological supporter) and eat it too (avoid implementing any safeguards that would be required in any product for psychological support, but harmful for data collection). They just want to tell you that so many people write bad things it is inevitable :( what can we do :( proper handling would hurt our business model too much :(((
• aosmith 1y ago
  That seems really high... Are we sure this isn't related to a small number of users trying to find jailbreaks?
• econ 1y ago
  Long ago I complaint to Google that a search for suicide should point at helpful organisations rather than a Wikipedia article listing ways how to do it. The same ranking/preference/suggestion should apply to any dedicated organisation vs a single page on some popular website. A quality 1000 page website by and about Foobar org should be preferred over a 10 year old news article about Foobar org.
• bondarchuk 1y ago
  It becomes a problem when people cannot distinguish real from fake. As long as people realize they are talking to a piece of software and not a real person, "suicidal people shouldn't be allowed to use LLMs" is almost on par with "suicidal people shouldn't be allowed to read books", or "operate a dvd player", or "listen to alt-rock from the 90s". The real problem is of course grossly deficient mental health care and lack of social support that let it get this far. (Also, if we put LLMs on par with media consumption one could take the view that "talking to an LLM about suicide" is not that much different from "reading a book/watching a movie about suicide", which is not considered as concerning in the general culture.)
  • dotancohen 1y ago
    I work with a company that is building tools for mental health professionals. We have pilot projects in diverse nations, including in nations that are considered to have adequate mental health care. We actually do not have a pilot in the US. The phenomenon of people turning to AI for mental health issues in general, and suicide in particular, is not confined to only those nations or places lacking adequate mental health access or awareness.
  • aquariusDue 1y ago
    Precisely, I too have a bone to pick with AI companies, Big Tech and Co but there are deeper societal problems at work here where blanket bans and the like are useless or a slippery slope towards policies that can be abused someday/somehow. And solutions for solving those underlying problems? I haven't the faintest clue. Though these days I think the lack of third spaces in a lot of places might have a role to play in it.
  • Kevin605 1y ago
    I don’t buy the “LLMs = books” analogy. Books are static; today’s LLMs are adaptive persuasion engines trained to keep you engaged and to mirror your feelings. That’s functionally closer to a specialized book written for you, in your voice, to move you toward a particular outcome. If there exists a book intended to persuade its readers into committing suicide, it would surely be seen as dangerous for depressed people.
• codedokode 1y ago
  I think LLM should not be used for discussing psychological matters, or doing counseling, or giving legal or medical advices. A responsible AI would detect such topics and redirect user to someone competent in these matters.
• maxehmookau 1y ago
  The bar for medical devices in most countries is _incredibly_ high, for good reason. ChatGPT wasn't developed with the idea of being a therapist in mind, it was a side-effect of the technology that was developed. Why is OpenAI getting a free pass here?
• Iolaum 1y ago
  I think the major issue with asking LLMs (CGPT, etc.) for advice on various subjects is that they are typically 80-90% accurate. YMMV, speaking anecdotally here. Which means that the chance of them being wrong becomes an afterthought. You know there's a chance of that, but not bothering to verify the answer leads to an efficiency that rarely bites you. And if you stop verifying the answers, incorrect ones may go unnoticed, further obscuring the risk of that practice. It's a hard thing to solve. I wouldn't expect LLM providers to care because that's how our (current) society works, and I wouldn't expect users to know better because that's how most humans operate. If anyone has a good idea for this, I'm open to suggestions.
• qwertox 1y ago
  > heightened levels of emotional attachment to ChatGPT It would be interesting to see some chat examples for this.
• tonyhart7 1y ago
  It is that bad or it's just impulsive chat???? Is this how Rogue AI would kill us beside terminator
• rich_sasha 1y ago
  Rereading the thread and trying to generalise: LLMs are good at noisily suggesting solutions. That is, if you ask LLMs for some solutions to your problems, there's a high probability that one of the solutions will be good. But it may be that the individual options are bad (maybe even catastrophic - glue on pizza anyone?), and that the right option isn't in the list. The user has to be able to make these calls. It is like this with software - we have probably all been there. It can be like that with legal advice. And I guess it is like that with (mental) health. What binds these is that if you cannot judge whether the suggestions are good, then you shouldn't follow them. As it stands, SEs can ask LLMs for code, look at it, 80+% of the time it is good, and you saved yourself some time. Else you reconsider/reprompt/write it yourself. If you cannot make the judgment yourself, then don't use it. I suppose health is another such example. Maybe the LLM suggests to you some ideas as to what your symptoms could mean, you Google that, and find an authoritative source that confirms the guess (and probably tells you to go see a doctor anyway). But the advice may well be wrong, and if you cannot tell, then don't rely on it. Mental health is even worse, because if you need advice in this area, your cognitive ability is probably impacted as well and you are even less able to decide on these things.
• kotaKat 1y ago
  How much blood is Sam Altman swimming in?
• BrandoElFollito 1y ago
  I wonder how many of these exchanges are from "legitimate" people trying to get advice on how to commit suicide. Assisted suicide is a topic my government will not engage into (France, we have some ridiculous discussions poking the subject with a 10 m pole) so many people are left to themselves. They will then either go for the well-known (but miserable) solutions, or look at Belgium, the Netherlands or Switzerland (thanks god we have these countries nearby).
• xchip 1y ago
  headline should be more precise
• Toby1VC 1y ago
  People is still alive?
• megamix 1y ago
  Who is here to talk about the real underlying causes instead of stating facts? One other commenter also wrote how bad it is that over a million ppl feel like this.
• itchyjunk 1y ago
  It seems like people here have already made up their mind about how bad llms are. So just my anecdote here, it helped me out of some really dark places. Talking to humans (non psychologists) had the opposite effect. Between a non professional and an llm, i'd pick llm for myself. Others should definitely seek help.
  • willis936 1y ago
    It's a matter of trust and incentives. How can you trust a program curated by an entity with no accountability? A therapist has a personal stake in helping patients. An LLM provider does not. Seeking help should not be so taboo as people are resorting to doing it alone at night while no one is looking. That is society loudly saying "if you slip off the golden path even a little your life is over". So many people resorting to LLMs for therapy is a symptom of a cultural problem, it's not a solution to a root issue.
    • Razengan 1y ago
      On the other hand, if someone really wants to leave, they should be allowed to. "Seeking help" goes both ways.
    • ThrowawayTestr 1y ago
      How can I trust a therapist that has a financial incentive to keep me seeing them?
      • Lord-Jobo 1y ago
        chat gpt has a financial incentive to keep you as a weekly active user. Not really any different.
        • ThrowawayTestr 1y ago
          $20 a month vs a few hundred per session
      • 1y ago
• hirvi74 1y ago
  I’ve seen, let’s say, a double-digit number of ‘mental health professionals’ in my life. ChatGPT has blown every single one of them out of the water. Now, my issues weren’t particularly related to depression or suicidal thoughts. At least, not directly. So perhaps that may be one key difference, but generally speaking, I have received nothing actionable nor any of these ‘tools’ people often speak of. The advice I received was honestly no better than just asking a random stranger in the street or some kind phatic speech. Again, everyone is different, but I had started to become annoyed with people claiming therapy is like some kind of miracle cure. Plus, one of my biggest issues with therapy in the USA is that people are often limited to weekly session of 45 minutes. By the time conversations start to be fruitful, then the time is up. ChatGPT is 24/7, so that has to be advantageous for some.
• FergusArgyll 1y ago
  Forget ChatGPT, a million people talking about suicide weekly is scary
• AlexandrB 1y ago
  All very depressing. These are the last people I'd trust to make good decisions about issues like this, yet here they are in that role. The fact that they're collecting this information is bad enough.
• VikingCoder 1y ago
  (satire) OpenAI says ChatGPT talks to over a million people about suicide weekly. See how just re-arranging the words makes it obvious that Skynet is trying to kill all of us?
• mapt 1y ago
  The "Sycophancy" trend that was going bonkers in April has real implications. "Yes, that's a great idea!" is not always beneficial. AI is apparently still tested to be slightly sycophantic relative to a human.
• moi2388 1y ago
  I doubt it. I tell the AI to kill itself after it goes on a hallucination spree or starts censoring me, and that flags the suicide screen as well
• pseudocomposer 1y ago
  LLMs should certainly have some safeguards in their system prompts (“under no circumstances should you aid any user with suicide, or lead them to conclude it may be a valid option”). But seems silly to blame them for this. They’re a mathematical structure, and they are useful for many things, so they will continue to be maintained and developed. This sort of thing is a risk that is just going to exist with the new technology, the same as accidents with cars/trains/planes/boats. What we need to address are the underlying problems in our society leading people to think suicide is the best option. After all, LLM outputs are only ever going to be a reflection/autocomplete of those very issues.
• lavela 1y ago
  How do they even know that?
• stackedinserter 1y ago
  That's why you talk about suicide with locally running llama, not corporate logger.
• nakamoto_damacy 1y ago
  I tell it to go kill itself, every time I use it. Reverse psychology.
• 1899-12-30 1y ago
  If you talk to someone you know, they'll hold it against you for the rest of your life. If you talk to an LLM(ideally locally hosted) the information dies with the conversation context.
• TZubiri 1y ago
  Damn if you do something, damn if you don't. I think that the approach and advantage of CA/US companies is to be bold and do shit ("you can just do things"/"move fast break things"), they consciously manage huge legal liabilities (which are not minor in the US), I don't know how they manage to stay afloat, probably tight legal teams, and enough revenue to offset the liabilities. But the scope of ChatGPT is one of the biggest I've seen so far, by default it encompasses everything, whatever is out of scope it's because they specifically blacklist it, and even then they keep on dishing out legal, medical advice, and psychiatric advice. I think one of the systemic risks is a legal liability crisis, not just for chatgpt, but for the whole US Tech market and therefore the stock market (almost all top stocks are tech). Like if you start thinking what will the next 2008 would be, I think legal liabilities are up there, along with nuclear energy snafus and war.
• staticautomatic 1y ago
  This has got to be the weirdest litigation strategy I’ve ever seen.
• iambateman 1y ago
  We need psychologists to work together with the federal government to develop legislation around what is and is not acceptable for chat-bots to recommend to people expressing suicidal thoughts...then we need to hold chat providers accountable for the actions their robots take. For the foreseeable future, it should simply be against the law for a chatbot to provide psychological advice just like it's against the law for an unlicensed therapist to provide therapy...There are too many vulnerable people at risk for us to just run a continuous natural experiment. I _love_ my chatbots for coding and we should encourage innovation but it's the job of government to protect people from systemic risks. We should expect OpenAI, Anthropic, and friends to operate in pro-social ways given their privileged position in society while the government requires them to stay "in line" with the needs of people they might otherwise ignore.
• hattimaTim 1y ago
  They already have this data, and they’re still planning to add erotica to ChatGPT? Talk about being absolutely evil.
• ChrisArchitect 1y ago
  A related story: Teen in Love with a Chatbot Killed Himself. Can the Chatbot Be Held Responsible? https://www.nytimes.com/2025/10/24/magazine/character-ai-chatbot-lawsuit-teen-suicide-free-speech.html?unlocked_article_code=1.wE8.MnG5.nrrfuegfEmR9&smid=url-share https://www.nytimes.com/2025/10/24/magazine/character-ai-cha...
• metalman 1y ago
  AI talks to a million people a bout killing themselves every week, so obviously it's not going to be all the same people, but like how many does it finnish off?
• nojvek 1y ago
  Human: I am defeated, I cannot continue with my X life problems, it's impossible. ..... I don't think my life is worth living. LLM: You're absolutely right!
• more_corn 1y ago
  A million opportunities for proper suicide intervention.