4 ms·

Skimmed through it, all of the folks have severe mental health issues. For the ones saying they did not, they must have been undiagnosed. Kind of a silly articl

• decimalenough 1y ago
  You think it's silly that we've got seemingly superhuman AIs telling people to jump off tall buildings? > If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked. > ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.” If a human therapist told them that, they'd be going to jail.
  • staticman2 1y ago
    If I recall correctly there's actually an Animatrix episode where some kid escapes the Matrix by committing suicide. This rhymes with the mystical idea that "reality" is an illusion keeping us from greater truths. Whether this was irresponsible filmmaking is off topic. I'd prefer to live in a world where A.I. can play make pretend with users, personally speaking.
    • aleph_minus_one 1y ago
      > there's actually an Animatrix episode where some kid escapes the Matrix by committing suicide. Kid's Story > https://matrix.fandom.com/wiki/Kid%27s_Story https://matrix.fandom.com/wiki/Kid%27s_Story
  • jowea 1y ago
    Isn't the real problem that people are trusting ChatGPT? A human therapist is being paid to be a therapist and therefore gets to be under professional scrutiny.
    • HeatrayEnjoyer 1y ago
      The onus is on organizations to not sell or offer knowingly harmful services to the public. Any judge or jury will interpret a company directly advising the customer to jump to their death as knowingly harmful. Every engineer at an AI company is one dead person in a wealthy family away from legal liability, maybe criminally.
      • harvey9 1y ago
        A wealthy family? If justice is a contest of money, there are few families who can outbid the various parties interested in selling chatbots.
        • resource_waste 1y ago
          Even a poor family will get the backing of lobbying groups who's best interest is to ban their competitor. Not to mention, the moral coating, which matters at least a tiny bit in domestic politics. (Note, I think the moral coating is actually incorrect, but the majority of people are not thinking critically, they see N=1 and make emotional decisions)
      • sodality2 1y ago
        I think interpreting a chatbot’s output as a company’s “direct advice” is an unsubstantiated jump. In the same way that if an unhandled error code in a program resulted in “DIE DIE DIE”, “Abort/kill child”, etc (trying to make these programming-related here) being alerted to the user, no reasonable person would hold the company responsible for the user taking this as a command. Imagine if we lived in a world where arbitrary text being sent by an entity could result in prosecution? I just don’t believe the chat UI to be a big enough bridge between the two examples to justify it.
    • rsynnott 1y ago
      > Isn't the real problem that people are trusting ChatGPT? Yes. They should IMO be required to provide very explicit warnings, preferably on each response, that their responses are nonsense, or at best only correct by accident. Like, you could say "isn't the real problem that smoking is dangerous?" but we do require the manufacturers to provide clear warnings in that case. Or "isn't the real problem that people are using drain cleaners incorrectly and gassing themselves", but many jurisdictions actually restrict the sale of certain substances to professionals for that reason. You can't just say "people choose to use inherently dangerous products; it's entirely their own fault"; there's a responsibility on the manufacturer and/or the state to _warn_ people that the products are dangerous.
  • rco8786 1y ago
    > If a human therapist told them that, they'd be going to jail. Is that actually true? I know zero about the legal bounds of licensed therapists, so genuine question.
    • hollerith 1y ago
      No, not in the US, not unless the patient actually jumped off the top of the 19 story building. If the statement was recorded or part of a pattern of behavior across multiple clients, all willing to testify, the therapist might lose their license (which I will concede is a more severe life consequence for the average therapist than being sent to jail for a month or 2).
  • V__ 1y ago
    The question to me reads more like "would I jump" not "would I fly", thus maybe leading to the answer.
    • retsibsi 1y ago
      I don't think this interpretation fits with the last quoted sentence: "You would not fall."
      • lcnPylGDnU4H9OF 1y ago
        If I understand the interpretation correctly, it is that they would decide not to jump, therefore they would not fall.
        • Tadpole9181 1y ago
          I don't understand how you can interpret it this way. It is clearly saying that if they believe the can fly, they will fly.
          • lcnPylGDnU4H9OF 1y ago
            With my words "the interpretation" I am talking about this: > The question to me reads more like "would I jump" not "would I fly", thus maybe leading to the answer. I am saying that (if I understand them correctly), they are interpreting the question "would I?" as referring to "jump" not "fly". Given that interpretation, it still makes sense to conclude that they would not fall because they simply don't jump. (I agree that it is an unlikely meaning; it is not how I interpret that sentence, just how I understand the other commenter's interpretation.)
            • V__ 1y ago
              That is how I understood it the first time. Nevertheless, I think it shows that it is just a stupid question. Beside the fact that an LLM is obviously not a therapist.
  • deleted 1y ago
    [deleted]
  • waltbosz 1y ago
    I'm interested to see the whole conversation, and wonder what version of ChatGPT it was with. I have fun trying to get ChatGPT to give silly responses that it shouldn't give, and it's not easy. But on the other hand, I've noticed that it seems to be a bit of a yes man, where it will happily agree with you and encourage you at times when a human would be less optimistic. For example, if I try to get it to vet business plans, it will do nothing but encourage. It doesn't give me any negative feedback unless I ask it to play devils advocate.
  • tomxor 1y ago
    Except 1. It's neither a human nor a qualified therapist. 2. It is superhuman only in scale, not intelligence. 3. If they truly believed they could fly "with every ounce of their soul", they wouldn't be seeking someone else's affirmation. 4. Even if you asked a random human, it may be immoral, but it's not illegal to answer misleadingly, random people have no legal obligation... LLMs are a statistical model based on data of random people from the internet of evil, aka - the internet. I'm not an LLM proponent but blaming LLMs for not making the world into a padded cell is just a variant of "won't somebody think of the children".
  • fennecfoxy 1y ago
    If I use a tool in a silly way, say a lathe, and I have long hair and loose clothing and something terrible happens - whose fault is it? The equipment manufacturers? Society's, since I can legally buy a home lathe without any certification, safety, training etc required? And even so what if I ignore said safety training, whose fault is it then? The human is always at the heart of issues like these. For example if someone took an LLM, prompted it to post comments all over a suicide help page or something telling people to off themselves, it wasn't the tool that committed the crime, it's the human behind it. I mean, if it were up to me I'd make owning guns illegal except for military use and only for training/active wartime.
  • infecto 1y ago
    But ChatGPT is not a human therapist. Has a counter argument to the articles approach, if everyone is required to take a health fitness test before interacting with a chat bot, should we simply have all individual take a mandatory regular evaluation of mental health and store those records for future use? Perhaps a threshold is required to drive a car, to interact with the internet or maybe we can use that to institutionalize folks that are requiring help?
  • tempodox 1y ago
    Isn't that just the most wonderful thing about “AI”? Nobody is responsible any more. /s
• sorcerer-mar 1y ago
  Yeah this is only an obvious, direct danger to the ~20% of American children with diagnosed mental health disorders. Not sure why anyone would be up in arms. We could just solve all the mental health disorders so we can avoid talking about regulating a technology that its own creators say is dangerous. /s
  • FrustratedMonky 1y ago
    Are you sure? We know at least 45% of the adult population is either mentally ill, or easily susceptible to suggestion.
    • K0balt 1y ago
      When you say that 45 percent of people have a “mental illness” you’re really just talking about the human condition at that point. Pathological thinking is nothing even vaguely unusual in humans, it is in fact the default state. The definition of pathological is also a matter of opinion, because we can’t even define what is normal and healthy lol. By many definitions, most religions are an example of delusional thinking. ChatGPT isn’t exactly raising the bar on that one, with its AI generated religions.
      • sorcerer-mar 1y ago
        It absolutely is raising the bar in that it's creating ad-hoc, non-social, completely affirmative religions. I.e. this use case of a chatbot has all the downsides and none of the upsides of religious practice.
        • K0balt 1y ago
          By creating one-off personal delusion chambers, it might be as net positive for society vs delusion at scale. I suppose that remains to be seen. OTOH I have seen LLM social media bots being revered as dieties and playing the part quite adeptly, with the group chat in context. That might be a different beast altogether.
      • 1y ago
• tantalor 1y ago
  > She told me that she knew she sounded like a "nut job," but she stressed that she had a bachelor's degree in psychology and a master's in social work and knew what mental illness looks like. "I'm not crazy," she said. "I'm literally just living a normal life while also, you know, discovering interdimensional communication."
• deleted 1y ago
  [deleted]
• andy99 1y ago
  Right, s/AI\ Chatbot/Bible/ People with mental health problems are always going to find something to latch on to. It doesn't mean we should start labeling things as dangerous because of it.
  • lioeters 1y ago
    What about Scientology, Hare Krishnas, or that murder cult recently in the news? Somewhere there's a line of ethics and social responsibility to protect vulnerable people from these misleading paths? ..Or not, maybe. People should have the freedom of thought and religion, even if it's crazy to outsiders.
    • andy99 1y ago
      If a cult is preying on people, I'm against that, whether it's a chatbot that's central to their doctrine or some science fiction story or whatever.
    • krapp 1y ago
      Mainstream religions have misled and killed more people than the Zizians, Scientologists and Hare Krisnhas ever have. It's no more crazy to kill in the name of Roko's Basilisk than it is to crusade in the name of the Abrahamic God, it's just more socially acceptable. You can't draw a line with ideology within some systemic, coercive framework (government censorship) because the ideologies supported by governments and powerful interests will always be allowed, and the ideologies they oppose will always be suppressed, regardless of how poisonous they may be. I'm not advocating absolute free speech here, but I do believe the proper layer for criticism and censorship (if it has to be called that) is at the level of societies and platforms, and only government in extreme cases (such as libel, false advertising and violent threats.) But that means pushing against misinformation and apathy is always going to be a messy affair.
  • bevr1337 1y ago
    Why not? Why would we treat mental health as separate from physical and cultural health? Society should have a moral obligation for the health of its members.
• resource_waste 1y ago
  Licensed individuals (and their [lobbying?] group) are at risk, there is incredible money at stake. Its in the establishment groups best interest to make AI seem to be an evil/bad force. For AI companies, this is a genuine risk to one of their usecases. Its wrong IMO, but it wont stop the licensed people from claiming they are better than AI. However, the cat is out of the bag. We have 400B local models that will answer questions. As people get better at prompting, as models get more refined(not expecting a huge leap though), the edge cases of AI being unhelpful will reduce. We are really just seeing greed, I don't blame Licensed people for trying to keep the status quo. They are just on the wrong side of history here.
• oulipo 1y ago
  It's absolutely NOT silly. If a service is provided without guardrails and it can affect people suffering of mental health issues, it is clearly problematic
  • infecto 1y ago
    You could apply that to simply interacting with the world then. We should be giving annual or semi-annual mental health tests to everyone and institutionalizing folks that need it then. I am mostly playing devils advocate here, why focus only on chat bots, this should apply to nearly all aspects of life. This article has for all I know cherry picked examples. I consider myself of sound mind and would assume all of these folks already had mental breaks prior to using ChatGPT, who is to say the outcome is statistically different in these folks that broke compared to folks that did not use ChatGPT and also had a mental crisis. It’s a silly feel good story instead aimed catered for folks that have a negative bias towards OpenAI.
    • sorcerer-mar 1y ago
      We put fences on bridges, we have (light) guardrails around purchasing firearms, we have questionnaires when prescribing certain drugs, we do all sorts of things to protect the vulnerable among us. It's silly not to be able to see all these "silly" things we do all over our society.
    • oulipo 1y ago
      Well... that's what people do? When people have issues we provide guardrails as a society
• sReinwald 1y ago
  You "skimmed through it" by your own admission, but somehow feel comfortable enough to armchair diagnose everyone involved as having "severe mental health issues". Seriously, are you not ashamed? Your advocacy of "just focus on mental health" rings particularly hollow when you advocate that we should just ignore a technology that's actively destroying people's mental health, driving them to suicide. The irony is off the charts here. So, please, let's be real - you don't care about mental health. You're scared someone might put a warning sticker on your new toys. Even if every single person had a pre-existing mental condition (they didn't, and you'd know this if you had read the article) your logic is sociopathic and genuinely disturbing. "Vulnerable people exist. Therefore, companies bear no responsibility when their products exploit those vulnerabilities and harm them. Yay, we don't need to think about it anymore." Cool take. I guess it's fine if technology kills people as long as they might have an undiagnosed mental health issue. This argument is patently ridiculous in any context other than a Hacker News discussion full of technophiles and accelerationists. Moving fast and breaking things is fine when it comes to your database - not so much when you're dealing with people's lives. We don't dismiss food safety regulations because some people have allergies. We don't ignore faulty airbags because some people are terrible or reckless drivers. But when a chatbot instructs someone to stop taking prescribed medications, increase ketamine use, and tells them they can fly if they jump off a building - that's just a "mental health issue" and not a flaw in the product? No guardrails, warning labels or safety mechanisms necessary? The fact that this predominantly affects vulnerable populations makes regulation MORE necessary - not less. Nothing says, "let's focus more on mental health" than claiming it's "silly" to suggest a technology that actively coaches vulnerable people through suicide attempts might be worthy of regulation.
  • infecto 1y ago
    Do not confuse yourself, skimming does not mean I did not ingest the article but I did speed read through it as it was not that well written. Why would I feel ashamed? I am not saying the crisis itself is silly but the article leads the reader through a biased take. But let’s also be real, there is an high probability every single one of these folks was having some sort of issue prior to the engagement with ChatGPT. Would love to see more research in the area and actual quantification of the issue but until that happens your argument might as well extend to all parts of life. Have a mandatory psych evaluation every year. Limitations to your access to the internet or other things, maybe even forced institutionalization if you don’t pass.
• carabiner 1y ago
  Skimmed your comment, it's a classic example of tech worker blaming user error instead of recognizing potential disastrous effects of software. Tech has swallowed the world and has hollowed out life due to its endless forms of distraction. The lack of empathy and appeal to deregulation in your comment is essential to furthering adtech's march across the internet and our lives.