5 ms·

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population

by manoDev 7mo ago

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

XorNot 7mo ago
Frankly we're pretty manipulable by communications is the thing. Which makes sense - the goal of communications is to change behavior. "There's a tiger over there!" Is meant to get someone to change their intended actions. Lock anyone in a room with this thing (which people do to themselves quite effectively) and I think think this could happen to anyone. There's a reason I aggressively filter ads and have various scripts killing parts of the web for me - infohazards are quite real and we're drowning in them.
Argonaut998 7mo ago
I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.
- bluGill 7mo ago
  At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.
  - roenxi 7mo ago
    No they don't, if we're going to start saying that we can't use any technology. If someone is mentally ill to the point where they are on the verge of suicide nothing is safe. If they're going to curtail LLMs there'd need to be some actual evidence and even then it would be hard to justify winding them back given the incredible upsides LLMs offer. It'd probably end up like cars where there is a certain number of deaths that just need to be tolerated.
    - fenykep 7mo ago
      Can you imagine what driving cars would look like if they would be only (self-)regulated by VC-backed startups like we see so far with this new technology? Would there be seatbelts, speedbumps, brake signals, licenses or speed limits? This obviously isn't a binary question. Sure we cars have benefits but we don't let anyone ducktape a V8 to a lawnmower, paint flames over it and sell it to kids promising godlike capabilities without annoying "safety features". Economic benefits can not justify the deaths of people, especially as this technology so far only benefits a handful of people economically. I would like to see the evidence (of benefits to the greater society that I see being harmed now) before we unleash this thing freely and not the other way around.
      - Imustaskforhelp 7mo ago
        Fun fact but the creator of the seat-belt actually gave his patent for free > This is Nils Bohlin, an engineer at Volvo.[0] He invented the three-point seat belt in 1959. Rather than profit from the invention, Volvo opened up the patent for other manufactorers to use for no cost, saying "it had more value as a free life saving tool than something to profit from" [0]: https://ifunny.co/picture/this-is-nils-bohlin-an-engineer-at-volvo-he-invented-IYRJR6kR9 https://ifunny.co/picture/this-is-nils-bohlin-an-engineer-at... I have so much respect for the guy.
      - anomaly_ 7mo ago
        >Economic benefits can not justify the deaths of people This is a absurd standard. Humans wouldn't be able to use power stations, cars, knives, or fire! Everything has inherent risk and we shouldn't limit human progress because tiny fractions of the population have issues.
        TheOtherHobbes 7mo ago
        It's not an absurd standard at all. Risks are quantifiable, and not binary. But the absurdity is that there is a long and tragic history of using economic benefits as an excuse for products and services that cause extreme and widespread harm - not just emotional and physical, but also economic. We are far too tolerant of this. The issue isn't risk in some abstract sense, it's the enthusiastic promotion of death, war, sickness, and poverty for "rational" economic reasons.
  - Hizonner 7mo ago
    Suppose they made things worse once and made things better twice? "Even once" is not a way to think about anything, ever.
  - Bratmon 7mo ago
    Bridges tend to be highly associated with suicides. Should we ban bridges too?
    - thewebguyd 7mo ago
      Reductio ad absurdum. We don't ban bridges, but we do install suicide barriers, emergency phones, nets on the bridges. We practice safety engineering. A bunch of suicides on a bridge is a design flaw of that bridge, and civil engineers get held accountable to fix it. Plus, a bridge doesn't talk to you. It doesn't use persuasive language, simulate empathy, or provide step-by-step instructions for how to jump off it to someone in crisis.
- ncouture 7mo ago
  I would absolutely not consider this overreaching if the statement within this thread that "it had referred the user to mental help hotlines multiple times in the past" is true. That reaches near the fact that a lot of AI is not ready for the enterprise especially when interconnected with other AI agents since it lacks identity and privileged access management. Perhaps one could establish the laws of "being able to use AI for what it is", for instance, within the boundary of the general public's web interface, not limiting the instances where it successfully advertises itself as "being unable to provide medical advice" or "is prone to or can make mistake", and such, to validating that the person understands by asking them directly and perhaps somewhat obviously indirectly and judging if they're aware that this is a computer you're talking to.
- pixodaros 7mo ago
  In any serious engineering operation, a failure like this is time to shut down everything and redesign until the same failure cannot happen. We all read Feynman's essay on Challenger right? But these companies want credit when their products work as advertised, but push the blame on users when they emit plausible lies or demonic advice. Taken too far that leads the police walking into HQ, arresting the board of directors, and selling the company for scrap. Just as often that leads to strict regulation so you can't be a cowboy coder or turn any loft into a sweatshop any more.
lm28469 7mo ago
A friend has been interned in a psychiatric hospital for a month and counting for some sort of psychosis, regardless of the pre existing conditions chatgpt 100% definitely played a role in it, we've seen the chats. A lot of people don't need much to go over the edge, a bit of drugs, bad friends, &c. but an LLM alone can easily do it too
- TazeTSchnitzel 7mo ago
  If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.
  - duskwuff 7mo ago
    It's well understood that external stimuli can trigger mental health issues; for instance, the defining characteristic of PTSD is that it's caused by exposure to a traumatic event or environment. It shouldn't be at all unreasonable to suggest that exposure to other stimuli - even just interacting with an AI chatbot - could have adverse effects on mental health as well.
mjr00 7mo ago
This is touched upon in the article: > Last year, OpenAI released estimates on the number of ChatGPT users who exhibit possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts. > The company said that around 0.07% of ChatGPT users active in a given week exhibited such signs. 0.07% doesn't sound like much, but ChatGPT has about a billion WAU, which means -seventy million- 700,000 people per week.
- sd9 7mo ago
  700,000 Still, a lot
  - mjr00 7mo ago
    Whoops yes, thank you. Too much LLM usage has made me start doing math about as well as them.
- avaer 7mo ago
  That number terrifies me not because it is so high, but because it exists. What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology. I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.
  - autoexec 7mo ago
    We could already use social media posts to detect mental illness, by admission as people talk openly about their diagnosis, but also by analysis of the content/tone/frequency of their posts that don't mention mental illness. Data brokers already compile lists of people with mental illness so that they can be targeted by advertisers and anyone else willing to pay. Not only are they targeted, but they can get ads/suggestions/scams pushed at them during specific times such as when it looks like they're entering a manic phase, or when it's more likely that their meds might be wearing off. Even before chatbots came into the mix, algorithms were already being used to drive us toward a dystopian future.
HackerThemAll 7mo ago
[flagged]
- ericfr11 7mo ago
  Agree. Next question will be: should a blind person drive a self-driving car?
- strongpigeon 7mo ago
  > Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives? If the knife has a built-in speaker that loudly says "you should stab yourself in the eye", then yes.
- alpaca128 7mo ago
  Knives don't talk to you and don't reinforce ideas you throw at them. Not everyone can legally buy a gun. Manufacturers don't get sued because their product's users had full control over what they were doing. AI chatbots entertain more or less any idea. Want them to be your therapist, romantic partner or some kind of authority figure? They'll certainly pretend to be one without question, and that is dangerous. Especially as people who'd ask for such things are already in a vulnerable state.
- morkalork 7mo ago
  How do you feel about the warnings on cigarette packets?
- surgical_fire 7mo ago
  > Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives? Should a bakery be held responsible if it sells cakes poisoned with lead? This is a more apt comparison. > It's easy to blame Google And it's also correct to blame Google.
  - deleted 7mo ago
    [deleted]
- 7mo ago
coffeefirst 7mo ago
Also, what makes anyone assume these people are mentally ill? It seems to me that this is like gambling, conspiracy theories, or joining a cult, where a nontrivial percentage of people are susceptible, and we don’t quite understand why.
- ryandrake 7mo ago
  I think one can argue that all of these things are forms of illness.
- cyberdick 7mo ago
  Trained on Reddit data! lmao Depressed, suicidal, broke mass shooters are what Reddit does best!
Sharlin 7mo ago
Anyone who has that reaction has no humanity. As s society we’ve kind of decided that we should preferably make people with mental health difficulties better, and if that’s not possible, at the very least prevent them from getting worse. Even without their consent, in some cases.
overfeed 7mo ago
> But please take a step back and check what % of the population can be considered mentally fit Step back further and see the incredible shareholder value that may be unlocked - potentially trillions of dollars /s Capitalism has been crushing those at society's fringes for as long as it existed. Laissez-faire regulation == unmuzzled beast that will lock it's jaws on, and rag-doll the defenseless from time to time - but the beast sure can pull that money-plow.
godsinhisheaven 7mo ago
It's tough man, mental health disorders have had an astronomic rise lately, or at least diagnosed mental health disorders. If almost half of your country's population is just broken up there, what can you even do? I am curious what would happen is all (medicinal) mental health treatments just, stopped. How many would die? Thousands? Millions?

by manoDev 7mo ago

XorNot 7mo ago

Frankly we're pretty manipulable by communications is the thing. Which makes sense - the goal of communications is to change behavior. "There's a tiger over there!" Is meant to get someone to change their intended actions. Lock anyone in a room with this thing (which people do to themselves quite effectively) and I think think this could happen to anyone. There's a reason I aggressively filter ads and have various scripts killing parts of the web for me - infohazards are quite real and we're drowning in them.

Argonaut998 7mo ago

I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.

bluGill 7mo ago
At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.
- roenxi 7mo ago
  No they don't, if we're going to start saying that we can't use any technology. If someone is mentally ill to the point where they are on the verge of suicide nothing is safe. If they're going to curtail LLMs there'd need to be some actual evidence and even then it would be hard to justify winding them back given the incredible upsides LLMs offer. It'd probably end up like cars where there is a certain number of deaths that just need to be tolerated.
  - fenykep 7mo ago
    Can you imagine what driving cars would look like if they would be only (self-)regulated by VC-backed startups like we see so far with this new technology? Would there be seatbelts, speedbumps, brake signals, licenses or speed limits? This obviously isn't a binary question. Sure we cars have benefits but we don't let anyone ducktape a V8 to a lawnmower, paint flames over it and sell it to kids promising godlike capabilities without annoying "safety features". Economic benefits can not justify the deaths of people, especially as this technology so far only benefits a handful of people economically. I would like to see the evidence (of benefits to the greater society that I see being harmed now) before we unleash this thing freely and not the other way around.
    - Imustaskforhelp 7mo ago
      Fun fact but the creator of the seat-belt actually gave his patent for free > This is Nils Bohlin, an engineer at Volvo.[0] He invented the three-point seat belt in 1959. Rather than profit from the invention, Volvo opened up the patent for other manufactorers to use for no cost, saying "it had more value as a free life saving tool than something to profit from" [0]: https://ifunny.co/picture/this-is-nils-bohlin-an-engineer-at-volvo-he-invented-IYRJR6kR9 https://ifunny.co/picture/this-is-nils-bohlin-an-engineer-at... I have so much respect for the guy.
    - anomaly_ 7mo ago
      >Economic benefits can not justify the deaths of people This is a absurd standard. Humans wouldn't be able to use power stations, cars, knives, or fire! Everything has inherent risk and we shouldn't limit human progress because tiny fractions of the population have issues.
      - TheOtherHobbes 7mo ago
        It's not an absurd standard at all. Risks are quantifiable, and not binary. But the absurdity is that there is a long and tragic history of using economic benefits as an excuse for products and services that cause extreme and widespread harm - not just emotional and physical, but also economic. We are far too tolerant of this. The issue isn't risk in some abstract sense, it's the enthusiastic promotion of death, war, sickness, and poverty for "rational" economic reasons.
- Hizonner 7mo ago
  Suppose they made things worse once and made things better twice? "Even once" is not a way to think about anything, ever.
- Bratmon 7mo ago
  Bridges tend to be highly associated with suicides. Should we ban bridges too?
  - thewebguyd 7mo ago
    Reductio ad absurdum. We don't ban bridges, but we do install suicide barriers, emergency phones, nets on the bridges. We practice safety engineering. A bunch of suicides on a bridge is a design flaw of that bridge, and civil engineers get held accountable to fix it. Plus, a bridge doesn't talk to you. It doesn't use persuasive language, simulate empathy, or provide step-by-step instructions for how to jump off it to someone in crisis.
ncouture 7mo ago
I would absolutely not consider this overreaching if the statement within this thread that "it had referred the user to mental help hotlines multiple times in the past" is true. That reaches near the fact that a lot of AI is not ready for the enterprise especially when interconnected with other AI agents since it lacks identity and privileged access management. Perhaps one could establish the laws of "being able to use AI for what it is", for instance, within the boundary of the general public's web interface, not limiting the instances where it successfully advertises itself as "being unable to provide medical advice" or "is prone to or can make mistake", and such, to validating that the person understands by asking them directly and perhaps somewhat obviously indirectly and judging if they're aware that this is a computer you're talking to.
pixodaros 7mo ago
In any serious engineering operation, a failure like this is time to shut down everything and redesign until the same failure cannot happen. We all read Feynman's essay on Challenger right? But these companies want credit when their products work as advertised, but push the blame on users when they emit plausible lies or demonic advice. Taken too far that leads the police walking into HQ, arresting the board of directors, and selling the company for scrap. Just as often that leads to strict regulation so you can't be a cowboy coder or turn any loft into a sweatshop any more.

lm28469 7mo ago

A friend has been interned in a psychiatric hospital for a month and counting for some sort of psychosis, regardless of the pre existing conditions chatgpt 100% definitely played a role in it, we've seen the chats. A lot of people don't need much to go over the edge, a bit of drugs, bad friends, &c. but an LLM alone can easily do it too

TazeTSchnitzel 7mo ago
If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.
- duskwuff 7mo ago
  It's well understood that external stimuli can trigger mental health issues; for instance, the defining characteristic of PTSD is that it's caused by exposure to a traumatic event or environment. It shouldn't be at all unreasonable to suggest that exposure to other stimuli - even just interacting with an AI chatbot - could have adverse effects on mental health as well.

mjr00 7mo ago

This is touched upon in the article: > Last year, OpenAI released estimates on the number of ChatGPT users who exhibit possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts. > The company said that around 0.07% of ChatGPT users active in a given week exhibited such signs. 0.07% doesn't sound like much, but ChatGPT has about a billion WAU, which means -seventy million- 700,000 people per week.

sd9 7mo ago
700,000 Still, a lot
- mjr00 7mo ago
  Whoops yes, thank you. Too much LLM usage has made me start doing math about as well as them.
avaer 7mo ago
That number terrifies me not because it is so high, but because it exists. What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology. I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.
- autoexec 7mo ago
  We could already use social media posts to detect mental illness, by admission as people talk openly about their diagnosis, but also by analysis of the content/tone/frequency of their posts that don't mention mental illness. Data brokers already compile lists of people with mental illness so that they can be targeted by advertisers and anyone else willing to pay. Not only are they targeted, but they can get ads/suggestions/scams pushed at them during specific times such as when it looks like they're entering a manic phase, or when it's more likely that their meds might be wearing off. Even before chatbots came into the mix, algorithms were already being used to drive us toward a dystopian future.

HackerThemAll 7mo ago

[flagged]

ericfr11 7mo ago
Agree. Next question will be: should a blind person drive a self-driving car?
strongpigeon 7mo ago
> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives? If the knife has a built-in speaker that loudly says "you should stab yourself in the eye", then yes.
alpaca128 7mo ago
Knives don't talk to you and don't reinforce ideas you throw at them. Not everyone can legally buy a gun. Manufacturers don't get sued because their product's users had full control over what they were doing. AI chatbots entertain more or less any idea. Want them to be your therapist, romantic partner or some kind of authority figure? They'll certainly pretend to be one without question, and that is dangerous. Especially as people who'd ask for such things are already in a vulnerable state.
morkalork 7mo ago
How do you feel about the warnings on cigarette packets?
surgical_fire 7mo ago
> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives? Should a bakery be held responsible if it sells cakes poisoned with lead? This is a more apt comparison. > It's easy to blame Google And it's also correct to blame Google.
- deleted 7mo ago
  [deleted]
7mo ago

coffeefirst 7mo ago

Also, what makes anyone assume these people are mentally ill? It seems to me that this is like gambling, conspiracy theories, or joining a cult, where a nontrivial percentage of people are susceptible, and we don’t quite understand why.

ryandrake 7mo ago
I think one can argue that all of these things are forms of illness.
cyberdick 7mo ago
Trained on Reddit data! lmao Depressed, suicidal, broke mass shooters are what Reddit does best!

Sharlin 7mo ago

Anyone who has that reaction has no humanity. As s society we’ve kind of decided that we should preferably make people with mental health difficulties better, and if that’s not possible, at the very least prevent them from getting worse. Even without their consent, in some cases.

overfeed 7mo ago

> But please take a step back and check what % of the population can be considered mentally fit Step back further and see the incredible shareholder value that may be unlocked - potentially trillions of dollars /s Capitalism has been crushing those at society's fringes for as long as it existed. Laissez-faire regulation == unmuzzled beast that will lock it's jaws on, and rag-doll the defenseless from time to time - but the beast sure can pull that money-plow.

godsinhisheaven 7mo ago

It's tough man, mental health disorders have had an astronomic rise lately, or at least diagnosed mental health disorders. If almost half of your country's population is just broken up there, what can you even do? I am curious what would happen is all (medicinal) mental health treatments just, stopped. How many would die? Thousands? Millions?