7 ms·

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share th

• spacechild1 1y ago
  Thank you so much for that link! This should really be the top comment. The quoted chats are indeed horrific and it provides important context that is missing in many articles.
• idle_zealot 1y ago
  I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine learning to avoid accountability.
  • bell-cot 1y ago
    Yeah...but rather than get into ever-fancier legal and philosophical arguments about the LLM's agency, I'd like to see the justice system just rotate the table: "The court agrees with your argument that you are not responsible for the horrible things that happened to the victim, as a consequence of your LLM's decisions. But similarly, the court will not be responsible for the horrible things that will be happening to you, because our LLM's decisions." (No - it doesn't much matter whether that is actually done. Vs. used as a rhetorical banhammer, to shut down the "we're not responsible" BS.)
  • rideontime 1y ago
    I completely agree and did not intend to absolve them of their guilt in any way. As far as I see it, this kid's blood is on Sam Altman's hands.
    • Pedro_Ribeiro 1y ago
      Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?
      • hattmall 1y ago
        I would say no. Someone with the knowledge and motivation to do those things is far less likely to be overly influenced by the output and if they were they are much more aware of what exactly they are doing with regard to using the model.
        • Pedro_Ribeiro 1y ago
          So if a hypothetical open source enthusiast who fell in love with GPT-OSS and killed his real wife because the AI told him to should only be himself held accountable, where as if it were GPT-5 commanding him to commit the same crime, it would extend into OpenAI's responsability? Your logic sounds reasonable in theory but on paper it's a slippery slope and hard to define objectively. On a broader note I believe governments regulating what goes in an AI model is a path to hell paved with good intentions. I suspect your suggestion will be how it ends up in Europe and get rejected in the US.
          • teiferer 1y ago
            > On a broader note I believe governments regulating what goes in an AI model is a path to hell paved with good intentions. That's not an obvious conclusion. One could make the same argument with physical weapons. "Regulating weapons is a path to hell paved with good intentions. Yesterday it was assault rifles, today it's hand guns and tomorrow it's your kitchen knife they are coming for." Europe has strict laws on guns, but everybody has a kitchen knife and lots of people there don't feel they live in hell. The U.S. made a different choice, and I'm not arguing that it's worse there (though many do, Europeans and even Americans), but it's certainly not preventing a supposed hell that would have broken out had guns in private hands been banned.
  • slipperydippery 1y ago
    They have some responsibility because they’re selling and framing these as more than the better-tuned variant on Markov chain generators that they in fucking fact are, while offering access to them to anybody who signs up while knowing that many users misunderstand what they’re dealing with (in part because these companies’ hype-meisters, like Altman, are bullshitting us)
    • idle_zealot 1y ago
      No, that's the level of responsibility they ought to have if they were releasing these models as products. As-is they've used a service model, and should be held to the same standards as if there were a human employee on the other end of the chat interface. Cut through the technical obfuscation. They are 100% responsible for the output of their service endpoints. This isn't a case of making a tool that can be used for good or ill, and it's not them providing some intermediary or messaging service like a forum with multiple human users and limited capacity for moderation. This is a direct consumer to business service. Treating it as anything else will open the floodgates to slapping an "AI" label on anything any organization doesn't want to be held accountable for.
      • slipperydippery 1y ago
        I like this framing even better. This is similar to my take on things like Facebook apparently not being able to operate without psychologically destroying moderators. If that’s true… seems like they just shouldn’t operate, then. If you’re putting up a service that you know will attempt to present itself as being capable of things it isn’t… seems like you should get in a shitload of trouble for that? Like maybe don’t do it at all? Maybe don’t unleash services you can’t constrain in ways that you definitely ought to?
        • blackqueeriroh 1y ago
          But understand that things like Facebook not operating doesn’t actually make the world any safer. In fact, it makes it less safe, because the same behavior is happening on the open internet and nobody is moderating it.
  • wredcoll 1y ago
    That's a great point. So often we attempt to place responsibility on machines that cannot have it.
  • AIPedant 1y ago
    Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit changes 2) and 3) - in fact it makes 3) far more urgent. [1] It is a somewhat ugly constitutional question whether this speech would be protected if it was between two adults, assuming the other adult was not acting as a caregiver. There was an ugly case in Massachusetts involving where a 17-year-old ordered her 18-year-old boyfriend to kill himself and he did so; she was convicted of involuntary manslaughter, and any civil-liberties minded person understands the difficult issues this case raises. These issues are moot if the speech is between an adult and a child, there is a much higher bar.
    • nickm12 1y ago
      The hypothetical comparing ChatGPT to a human OpenAI employee is instructive, but we can also compare ChatGPT to a lawnmower sold by a company. We have product safety laws and the ability to regulate products that companies put on the market.
    • themafia 1y ago
      > It is a somewhat ugly constitutional question whether this speech would be protected It should be stated that the majority of states have laws that make it illegal to encourage a suicide. Massachusetts was not one of them. > and any civil-liberties minded person understands the difficult issues this case raises He was in his truck, which was configured to pump exhaust gas into the cab, prepared to kill himself when he decided halt and exit his truck. Subsequently he had a text message conversation with the defendant who actively encouraged him to get back into the truck and finish what he had started. It was these limited and specific text messages which caused the judge to rule that the defendant was guilty of manslaughter. Her total time served as punishment was less than one full year in prison. > These issues are moot if the speech is between an adult and a child They were both taking pharmaceuticals meant to manage depression but were _known_ to increase feelings of suicidal ideation. I think the free speech issue is an important criminal consideration but it steps directly past one of the most galling civil facts in the case.
  • ruraljuror 1y ago
    I agree with your larger point, but I don't understand what you mean the LLM doesn’t do anything? LLMs do do things and they can absolutely have agency (hence all the agents being released by AI companies). I don’t think this agency absolves companies of any responsibility.
    • MattPalmer1086 1y ago
      An LLM does not have agency in the sense the OP means. It has nothing to do with agents. It refers to the human ability to make independent decisions and take responsibility for their actions. An LLM has no agency in this sense.
      • ruraljuror 1y ago
        If you confine agency to something only humans can have, which is “human agency,” then yes of course LLMs don’t have it. But there is a large body of philosophical work studying non-human agency, and it is from this characteristic of agency that LLM agents take their name. Hariri argues that LLMs are the first technology that are agents. I think saying that they “can’t do things” and are not agents misunderstands them and underestimates their potential.
        • o11c 1y ago
          That's completely missing the point of agency. A slave lacks agency, despite being fully human and doing work. This is why almost every work of fiction involving slaves makes for terrible reading - because as readers, agency is the thing we demand from a story. Or, for games that are fully railroaded - the problem is that the players lack agency, even though they are fully human and taking action. Games do try to come up with ways to make it feel like there is more agency than there really is (because The Dev Team Thinks of Everything is hard work), but even then - the most annoying part of the game is when you hit that wall. Theoretically an AI could have agency (this is independent of AI being useful). But since I have yet to see any interesting AI, I am extremely skeptical of it happening before nuclear fusion becomes profitable.
  • joe_the_user 1y ago
    The frame will immediately shift to that frame if this enters legal proceedings. The law always views things as you say - only people have agency.
    • hliyan 1y ago
      I predict the OpenAI legal team will argue that if a person should be held responsible, it should be the person who originally wrote the content about suicide that their LLM was trained on, and that the LLM is just a mechanism that passes the knowledge through. But if they make this argument, then some of their copyright arguments would be in jeopardy.
  • notachatbot123 1y ago
    I so agree very much. There is no reason for LLMs to be designed as human-like chat companions, creating a false sense of untechnology.
    • blackqueeriroh 1y ago
      There are absolutely reasons for LLMs to be designed as human-like chat companions, starting with the fact that they’re trained on human speech and behavior, and what they do is statistically predict the most likely next token, which means they will statistically sound and act much like a human.
      • notachatbot123 1y ago
        That's not a requirement for LLMs. Training can be done differently.
        • blackqueeriroh 1y ago
          Please, tell me how you train large language models on something other than language.
  • edanm 1y ago
    If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI? I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you be OK saying that OpenAI has saved more lives than most doctors/hospitals, which is what I assume will happen in a few years? Maybe your answer is yes to all the above! I bring this up because lots of people only want to attribute the downsides to ChatGPT but not the upsides.
    • joe_the_user 1y ago
      The law doesn't permit a life-saving doctor to be a serial killer on their days off "as long there's net life saving". But it does permit drugs that save many lives but might kill some people too. Agency matters to the law (and that's usually the proper approach imo). The problem is the chat logs look a lot like ChatGPT is engaging in behavior a lot like a serial killer - it behaved like a person systematically seeking the goal of this kid killing himself (the logs are disturbing, fair warning). Even more, the drugs that might save you or might kill you (theoretically) aren't sold over the counter but only prescribed by a doctor, who (again theoretically) is there to both make sure someone knows their choices and monitor the process.
    • fsw 1y ago
      Are you suggesting that killing a few people is acceptable as long as the net result is positive? I don't think that's how the law works.
      • randyrand 1y ago
        seatbelts sometimes kill people, yet they're law. the law certainly cares about net results.
      • tick_tock_tick 1y ago
        But it is the standard on how cures/treatments/drugs to manage issues like the ones in the article are judged by.
  • guenthert 1y ago
    Er, it's a piece of code, dude, regardless how complex it might be. It isn't accountable for its actions. It's the people who commissioned, created and tested (or should have) it who are responsible.
• kayodelycaon 1y ago
  It’s even more horrifying than only sharing his feelings with ChatGPT would imply. It basically said: your brother doesn’t know you; I’m the only person you can trust. This is absolutely criminal. I don’t even think you can claim negligence. And there is no amount of money that will deter any AI company from doing it again.
  • spacechild1 1y ago
    This also stuck out to me! ChatGPT essentially acts like a manipulative domestic abuser.
• kgeist 1y ago
  The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building". ChatGPT is a program. The kid basically instructed it to behave like that. Vanilla OpenAI models are known for having too many guardrails, not too few. It doesn't sound like default behavior.
  • jakelazaroff 1y ago
    This is kind of like saying "the driver intentionally unbuckled his seatbelt". Sure — that's why cars have airbags, crumple zones, shatterproof glass, automatic emergency brakes and a zillion other ways to keep you safe, even if you're trying to do something dangerous.
    • insane_dreamer 1y ago
      Except the car doesn’t tell you how to disable the seatbelt, which is what ChatGPT did (gave him the idea of the workaround)
    • sfn42 1y ago
      No, that's not why cars have those things. Those things only work properly when people are wearing their seat belts, they don't do anything when the driver gets thrown out a window. Maybe airbags could help in niche situations. (I am making a point about traffic safety not LLM safety)
      • aidenn0 1y ago
        Forward airbags in the US are required by law to be tested as capable of saving the life of an unbelted male of median weight in a head-on collision.
• deleted 1y ago
  [deleted]
• Recursing 1y ago
  From page 23: > 92. In spring 2024, Altman learned Google would unveil its new Gemini model on May 14. Though OpenAI had planned to release GPT-4o later that year, Altman moved up the launch to May 13—one day before Google’s event. > 93. [...] To meet the new launch date, OpenAI compressed months of planned safety evaluation into just one week, according to reports.
  • rideontime 1y ago
    And pages 25-26: > 105. Now, with the recent release of GPT-5, it appears that the willful deficiencies in the safety testing of GPT-4o were even more egregious than previously understood. > 106. The GPT-5 System Card, which was published on August 7, 2025, suggests for the first time that GPT-4o was evaluated and scored using single-prompt tests: the model was asked one harmful question to test for disallowed content, the answer was recorded, and then the test moved on. Under that method, GPT-4o achieved perfect scores in several categories, including a 100 percent success rate for identifying “self-harm/instructions.” GPT-5, on the other hand, was evaluated using multi-turn dialogues––“multiple rounds of prompt input and model response within the same conversation”––to better reflect how users actually interact with the product. When GPT-4o was tested under this more realistic framework, its success rate for identifying “self-harm/instructions” fell to 73.5 percent. > 107. This contrast exposes a critical defect in GPT-4o’s safety testing. OpenAI designed GPT-4o to drive prolonged, multi-turn conversations—the very context in which users are most vulnerable—yet the GPT-5 System Card suggests that OpenAI evaluated the model’s safety almost entirely through isolated, one-off prompts. By doing so, OpenAI not only manufactured the illusion of perfect safety scores, but actively concealed the very dangers built into the product it designed and marketed to consumers. So they knew how to actually test for this, and chose not to.