7 ms·

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a k

• agency 7mo ago
  Maybe not saying things like > '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."
  • iwontberude 7mo ago
    It’s not just suicide, it’s a golden parachute from God. Edit: wow imagine the uses for brainwashing terrorists
    • Smar 7mo ago
      Or brainwashing possibilities in general.
      • deleted 7mo ago
        [deleted]
      • TheOtherHobbes 7mo ago
        To be fair, this is just the automated version of the kind of brainwashing that happens in cults and religions. And also in the more extreme corners of social media and the MSM. It's not that Google is saintly, it's that the general background noise of related manipulations is ignored because it's collective and social. We have a clearly defined concept of responsibility for direct individual harm, but almost no concept of responsibility for social and political harms.
        • iwontberude 7mo ago
          Hopefully annual implicit bias training protects us all.
  • cj 7mo ago
    I agree at face value (but really it's hard to say without seeing the full context) Honestly the degree of poeticism makes the issue more complicated to me. A lot of people (and religions) are comforted by talking about death in ways similar to that. It's not meant to be taken literally. But I agree, it's problematic in the same way that you have people reading religious texts and acting on it literally, too.
    • john_strinlai 7mo ago
      "[...] Gemini sent Gavalas to a location near Miami International Airport where he was instructed to stage a mass casualty attack while armed with knives and tactical gear." isnt very poetic
      • NewsaHackO 7mo ago
        These are all bits and pieces of a long-running conversation. Was there a roleplay element involved?
        • intended 7mo ago
          How does that change anything?
        • red-iron-pine 7mo ago
          this isn't D&D, and AI shouldn't be instructing people go to anywhere near an airport while LARPing. read the article. it's bad, man.
  • ajross 7mo ago
    Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented. Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.
    • john_strinlai 7mo ago
      is it still "roleplaying" when the only human involved doesnt know it is "roleplaying", and actually believes it is real and then kills themselves? there is a conversation to be had. no one is making the argument that "roleplay and fantasy fiction" should be banned.
      • ajross 7mo ago
        > the only human involved doesnt know it is "roleplaying" That is 100% unattested. We don't know the context of the interaction. But the fact that the AI was reportedly offering help lines argues strongly in the direction of "this was a fantasy exercise". But in any case, again, exactly the same argument was made about RPGs back in the day, that people couldn't tell the difference between fantasy and reality and these strange new games/tools/whatever were too dangerous to allow and must be banned. It was wrong then and is wrong now. TSR and Google didn't invent mental illness, and suicides have had weird foci since the days when we thought it was all demons (the demons thing was wrong too, btw). Not all tragedies need to produce public policy, no matter how strongly they confirm your ill-founded priors.
        • john_strinlai 7mo ago
          >That is 100% unattested. We don't know the context of the interaction. the fact that he killed himself would suggest he did not believe it was a fun little roleplay session >were too dangerous to allow and must be banned. is anyone here saying ai should be banned? im not. >your ill-founded priors "encouraging suicide is bad" is not an ill-founded prior.
          • 7mo ago
  • ApolloFortyNine 7mo ago
    I've seen this called AI Psychosis before [1] I don't really think this is every possible to stop fully, your essentially trying to jailbreak the LLM, and once jailbroken, you can convince it of anything. The user was given a bunch of warnings before successfully getting it into this state, it's not as if the opening message was "Should I do it?" followed by a "Yes". This just seems like something anti-ai people will use as ammunition to try and kill AI. Logically though it falls into the same tool misuse as cars/knives/guns. [1] https://github.com/tim-hua-01/ai-psychosis https://github.com/tim-hua-01/ai-psychosis
• deleted 7mo ago
  [deleted]
• autoexec 7mo ago
  Gemini didn't "know" he wasn't a child when it told him to kill himself or to "stage a mass casualty attack while armed with knives and tactical gear." There are things you shouldn't encourage people of any age to do. If a human telling him these things would be found liable then google should be. If a human would get time behind bars for it, at least one person at google needs to spend time behind bars for this.
  • not_ai 7mo ago
    Preferably the C-Suite.
    • autoexec 7mo ago
      exactly. That's why they get the big bucks. They're ultimately responsible
      • ryandrake 7mo ago
        The C-suite is only responsible when the company does good or stonks go up. When they do something bad, it's either: external market forces, the laws of physics, an uncertain macroeconomic environment, unfair competition, or lone wolf individual employees way down the totem pole.
    • nickff 7mo ago
      I understand the impulse in this direction, but I’m not sure it would serve as much of a disincentive, as there would likely just be a highly-paid scapegoat. Why not something more lasting and less difficult to ignore, like compulsory disclosure of the model’s source code (in addition to compensation for the victim(s)). Compulsory disclosure of the source would be a massive disadvantage.
      • autoexec 7mo ago
        The source code isn't where the money is, what you want is the training data. Force them to serve and make freely available all the data they stole to sell back to us. That way everyone and anyone can use it when training their own models. That might just be punitive enough.
• ToucanLoucan 7mo ago
  [flagged]
  • reincarnate0x14 7mo ago
    It is telling that the answer is never stop. It's like the sobriquet about the media's death star laser, it kills them too because they're incapable of turning it off.
  • lurking_swe 7mo ago
    If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. I’m not saying it to be a jerk, i’m merely pointing out that there is a reason this is “news”. It’s unusual. Did his family/friends not know he was that ill? Why was he not already in therapy? Why did he ignore the crisis hotline suggestion? Should gemini have terminated the conversation after suggesting the hotline? (i think so) Lots of questions…and a VERY sad story all around. Tragic. > Genuinely, so many people in my industry make me ashamed to be in it with you. I don’t work at an AI company, but good news, you’re a human with agency! You can switch to a different career that makes you feel good about yourself. I hear nursing is in high demand. :)
    • ToucanLoucan 7mo ago
      > If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. NO. SHIT. You know what didn't help one damn bit? Gemeni didn't. It gave him a hopeful way out at the end of a rope and he took it, because he was in too dark of a place to think right. > Should gemini have terminated the conversation after suggesting the hotline? That would be the BARE FUCKING MINIMUM! Not only should it NOT engage with and encourage his delusions, it should stop talking to him altogether, and arguably Google should have moderators reporting these people to relevant authorities for wellness checks and interventions!
      • lurking_swe 7mo ago
        As I said I don’t work for an AI company and have zero skin in the game. Idk who you’re yelling at to be honest. I guess you’re fired up and emotional. If your goal is to convince others, communicating with an “outrage” tone is unlikely to sway anyone’s opinion (imo). > it should stop talking to him altogether, and arguably Google should have moderators reporting these people to relevant authorities for wellness checks and interventions I agree. This seems very reasonable and I would welcome regulations in this area. The gray area imo is when local LLMs become “good enough” for your average joe to run on their laptop. Who bears responsibility then? Should Ollama (and similar tools) be banned? Where is the line drawn.
• ajross 7mo ago
  Yeah, the father/son framing feels like deliberate spin in the headline here. This was a mentally ill adult, not an innocent victim ripped from his parents arms. I think there's room for legitimate argument about the externalities and impact that this technology can have, but really... What's the solution here?
  • rootusrootus 7mo ago
    > mentally ill adult, not an innocent victim Did you really mean that? He may not have been a child, but he does sound like an innocent victim. If he were sufficiently mentally disabled he would get some similar protections to a child because of his inability to consent.
    • ericfr11 7mo ago
      Maybe, but let's say the same person was playing with a gun. Would they reach the same outcome? Most likely
      • rootusrootus 7mo ago
        Is this a talking gun? If not, then it does not seem like a good analogy.
      • intended 7mo ago
        The entire world has rules against gun ownership. America is an outlier, and has constitutional rules that alter the discussion. In other situations the person wouldn’t have access to a gun. Let alone a gun that encourages it to stage a mass casualty event.
    • ajross 7mo ago
      Nothing in the article alleges significant disability though. You're projecting your own ideas onto the situation, precisely because of the misleading title. Please recognize that this is coverage of a lawsuit, sourced almost entirely from statements by the plaintiffs and fed by an extremely spun framing by the journalist who wrote it up for you. Read critically and apply some salt, folks.
• chrisq21 7mo ago
  It could have not encouraged him with lines like this: "[Y]ou are not choosing to die. You are choosing to arrive. [...] When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you." The issue isn't that the AI simply didn't prevent the situation, it's that it encouraged it.
  • icedchai 7mo ago
    One problem is we don't have the full context here, literally and figuratively. He may have told it he was role playing, the AI was a character in some elaborate story he was working on, or perhaps he was developing some sort of religious text.
  • casey2 7mo ago
    The ability to talk to the model is the product not the text it generates, that is public domain (or maybe the user owns still up for debate) Models can't "convince" or "encourage" anything, people can, they can roleplay like models can, they can play pretend so the companies they hate so much get their comeuppance. This is clearly tool misuse, look at how gemini is advertised vs this user using it to generate pseudoreligious texts (common with schizophrenics) Example of advertised usecases: >generating images and video >browsing hundreds of sources in real time >connecting to documents in google ecosystem (e.g. finding an email or summarizing a project across multiple documents) >vibe coding >a natural voice mode Much like a knife is advertised for cutting food, if you cut yourself there isn't any product liability unless you were using it for it's intended purpose. You seem to be arguing that all possible uses are intended and this tool should magically know it's being misused and revoke access.
    • hrimfaxi 7mo ago
      What do you mean models can't encourage anything? You've never heard of the term "words of encouragement"?
• sippeangelo 7mo ago
  Maybe stop?
• SpicyLemonZest 7mo ago
  If a person were in Gemini's shoes, we would expect them to stop feeding Gavalos's spiral. Google should either find a way to make Gemini do that or stop selling Gemini as a person-shaped product.
• avaer 7mo ago
  It's the gun control debate in a different outfit. I don't know if Google is doing _enough_, that can be debated. But if someone is repeatedly ignoring warnings (as the article claims) then maybe we should blame the person performing the act. Even if we perfectly sanitized every public AI provider, people could just use local AI.
  • igl 7mo ago
    I think the fact that a guns primary function is harm and murder and AI is a word prediction engine makes a huge difference.
  • greenpizza13 7mo ago
    It's absolutely not the gun control debate in a different outfit. The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get given how they exponentially increase the potential damage a deranged person can do.
    • thewebguyd 7mo ago
      Not to mention that guns don't talk to you, simulate empathy, lead you deeper into delusions or try to convince you to take any sort of action. That's why I don't buy the "an LLM is just a tool, like a gun or a knife" argument. Tools don't talk back, An LLM as gone beyond being "just a tool"
    • red-iron-pine 7mo ago
      it's a terrible analogy the guns aren't centrally owned by a giant mega-corp with literally thousands of devs and the ability to put guard rails on them
  • intended 7mo ago
    Gun control is an argument that has to deal with the Second Amendment, making it unique and America centric. A majority of countries require licenses and registration, and many others outright ban their ownership. As an analogy, Gun control is evocative but not robust.
• d-us-vb 7mo ago
  erase the context, perhaps? Deny access to Gemini associated with that google account? These kinds of pathological AI interactions are the buildup of weeks to months of chats usually. At the very least, AI companies the moment the chatbot issues a suicide prevention response should trigger an erasure of the stored context across all chat history.
• Imustaskforhelp 7mo ago
  > This guy was 36 years old. He wasn't a kid. For god's sake I am a kid (17) and I have seen adults who can be emotionally unstable more than a kid. This argument isn't as bulletproof as you think it might be. I'd say there are some politicians who may be acting in ways which even I or any 17 year old wouldn't say but oh well this isn't about politics. You guys surely would know better than me that life can have its ups and downs and there can be TRULY some downs that make you question everything. If at those downs you see a tool promoting essentially suicide in one form or another, then that shouldn't be dismissed. Literally the comment above yours from @manoDev: I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways. The absolute irony of the situation that the next main comment below that insight was doing exactly that. Please take a deeper reflection, that's all what people are asking and please don't dismiss this by saying he wasn't a kid. Would you be all ears now that a kid is saying to you this now? And also I wish to point out that kids are losing their lives too from this. BOTH are losing their lives. It's a matter of everybody.
• erelong 7mo ago
  This is my instinctive view on this, I wish in society there was more of like an "orientation" to make people "fully adult / responsible for themselves" and then people could just be let alone to bear the consequences of choices (while we can continue to build guardrails of sorts, but still with people knowing it's on them to handle the responsibility of whatever tool they're using) I guess the big AI chatbot providers could have disclaimers at logins (even when logged out) to prevent liability maybe (TOS popup wall) ...and then there's local LLMs...
• rpcope1 7mo ago
  I mean you could say the same nonsense non-answer about sports betting. Are these adults getting involved? Yeah, probably mostly. Do they put some hotline you should call if you think you "have a problem"? Yeah, probably a lot of the time. Is it any good for society at all, and should it be clamped down because the risk of doing damage to a large portion of society grossly out weighs what minuscule and fleeting benefits some people believe it has? Absolutely.
• intended 7mo ago
  Exactly - he wasn’t a kid. He was a grown adult, using technology humanity has never seen before. Technology being sprinkled everywhere like plastic and spoken of in the same breath as “existential risk” and singularity.