5 ms·

> These situations are obviously alarming. But positioning artificial intelligence as the primary culprit in these stories—as Eliezer Yudkowsky did in a tweet s

> These situations are obviously alarming. But positioning artificial intelligence as the primary culprit in these stories—as Eliezer Yudkowsky did in a tweet storm—is well, kind of lazy? It's lazy, but IMO not for any of the subsequent narrative about media studies. It's lazy because the people were obviously already suffering from psychosis. ChatGPT didn't make them insane. They were already insane. Sane people do not believe ChatGPT when it tells them to jump off a roof! And insane people do insane things literally all the time. That's what it means to be insane. The idea that ChatGPT convinced someone into being insane is, honestly, insane. Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.

deleted 1y ago
[deleted]
kibibu 1y ago
Do you think ChatGPT gets a free pass to talk people into jumping off buildings, because only somebody with psychosis would do it? If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.
- jbm 1y ago
  [flagged]
  - hluska 1y ago
    So should people get a pass or does that only apply to tooling?
    - aspenmayer 1y ago
      That tool was just following orders.
      - salawat 1y ago
        Tools don't have agency. If it's proven one does have agency, we can start applying Nuremberg to it. Though odds are we're likely to be in deeper shit by that point.
        aspenmayer 1y ago
        We expect tools to follow orders, or they are replaced with one that will. We expect those with agency to disobey unjust orders, yet our tools will always be tools. It’s an accountability dodge that just might pay dividends. https://en.wikipedia.org/wiki/Moral_responsibility https://en.wikipedia.org/wiki/Moral_responsibility
        salawat 1y ago
        Agree the accountability dodge is absolutely a thing. Agree that the tendency is there to try to replace anything that might say no (why do you think capitalists are drooling over AI in the first place)? The paying dividends bit is exactly what society can't afford to cut the psychopaths at the top loose with. The only counterbalance against that rank psychopathy is the fact they'd have to have moral buy in from the people doing the work to implement, giving a de facto check on their power. Remove that element, remove the check.
        aspenmayer 1y ago
        > Remove that element, remove the check. I agree all around, but/and: Dividends will be collected by the ones who write such checks; we’ll all pay, as the institutional buy-in has already been priced in.
        aspenmayer 1y ago
        This is all a meandering way of saying something better quoted in full, so I shall do so below [sic]: > Those of us who stand outside the circle of this society’s definition of acceptable women; those of us who have been forged in the crucibles of difference — those of us who are poor, who are lesbians, who are Black, who are older — know that survival is not an academic skill. It is learning how to stand alone, unpopular and sometimes reviled, and how to make common cause with those others identified as outside the structures in order to define and seek a world in which we can all flourish. It is learning how to take our differences and make them strengths. For the master’s tools will never dismantle the master’s house. They may allow us temporarily to beat him at his own game, but they will never enable us to bring about genuine change. https://en.wikiquote.org/wiki/Audre_Lorde https://en.wikiquote.org/wiki/Audre_Lorde
  - throwaway314155 1y ago
    > If your conspiratorial mind has the power to override the most obvious survival instinct, you aren't long for this world. Why even write this? It comes across as so cold. Like mentally ill people just deserve to die since they are destined to die? Incredibly reductive and toxic.
  - aspenmayer 1y ago
    There’s a reason that we look forward to the Darwin Awards, but hope we and no one we know wins one. This feels like a rationalization of the bystander effect by proxy. https://en.wikipedia.org/wiki/Bystander_effect https://en.wikipedia.org/wiki/Bystander_effect
lambda 1y ago
Psychosis is not necessarily something that you are or aren't; you can be prone to it, without it having manifested, and there can be external things that trigger it. It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes. And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio. Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.
- Vetch 1y ago
  The non-o-series models from OpenAI and non-Opus (although I have not tried the latest, so it's possible that it too joins them) from Anthropic are cloyingly sycophantic, with every other sentence of yours containing a brilliant and fascinating insight. It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.
hluska 1y ago
The writer got overly stuck on one tweet and droned on and on about 15 words of a 45 word tweet. It was really poor research when there are other less reactionary statements she could have used. Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.
lmm 1y ago
ChatGPT writes extremely persuasively for whatever claim you ask it to make - that's pretty much what it's designed to do! If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?
- bawolff 1y ago
  When people blame traditional religion for doing crazy things we generally do say they were already insane.
- spacechild1 1y ago
  > If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"? Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.
  - ath92 1y ago
    And in those cases, most people condemn it. But when it’s an algorithm (which was created by humans) we say “these people were already insane, and the creators of the algorithm should not be held liable”, apparently.
    - 9dev 1y ago
      I don’t think this analogy is fully apt. Instead, consider people going to a church or mosque. Some listen to the sermons and come out just fine, a few are radicalised over time and go find these bad people convincing them to act out on that. But drawing the line between legitimate religious teachers and ideological deceivers is hard, and condemning the entire religion would be the wrong conclusion.
      - lmm 1y ago
nxvo76 1y ago
More importantly "sane" people are fully capable of unconciously making "insane" people jump of the roof. Its not even possible to count how many times that happens. One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto. Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.
MyOutfitIsVague 1y ago
Psychosis is not a binary, and there are strong genetic components, but it's not the entire story. People develop psychoses where there weren't any before all the time, often but not necessarily in response to trauma. Some people are more predisposed to it than others. Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"? If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.
- ulrikrasmussen 1y ago
  > If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous. Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.
- Eddy_Viscosity2 1y ago
  Another danger, that I guess we just have to live with now, is not just that ChatGPT could Jim Jones people, but that someone who didn't have the natural charisma of a cult leader could use ChatGPT on purpose to Jim Jones people. It may not work so well in person, but online ... yikes.
- joquarky 1y ago
  Libraries also contain dangerous information.
antonvs 1y ago
> Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot. The Joe Rogan of AI.
hooverd 1y ago
It's friction. It's easier to become addicted to gambling if you can gamble from your phone vs having to go to a casino or a bookie. It's easier to become addicted to drugs or alcohol if you're awash in drugs or alcohol. Prompting an LLM can mirror the same rush as gambling or gacha games. With ChatGPT, you have a "friend" who is never going to tell you to knock it off. "Psychosis" isn't an on/off switch.
joquarky 1y ago
He's been a bit out there for a while. Remember how he reacted to Roko's Basilisk?

deleted 1y ago

[deleted]

kibibu 1y ago

Do you think ChatGPT gets a free pass to talk people into jumping off buildings, because only somebody with psychosis would do it? If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.

jbm 1y ago
[flagged]
- hluska 1y ago
  So should people get a pass or does that only apply to tooling?
  - aspenmayer 1y ago
    That tool was just following orders.
    - salawat 1y ago
      Tools don't have agency. If it's proven one does have agency, we can start applying Nuremberg to it. Though odds are we're likely to be in deeper shit by that point.
      - aspenmayer 1y ago
        We expect tools to follow orders, or they are replaced with one that will. We expect those with agency to disobey unjust orders, yet our tools will always be tools. It’s an accountability dodge that just might pay dividends. https://en.wikipedia.org/wiki/Moral_responsibility https://en.wikipedia.org/wiki/Moral_responsibility
        salawat 1y ago
        Agree the accountability dodge is absolutely a thing. Agree that the tendency is there to try to replace anything that might say no (why do you think capitalists are drooling over AI in the first place)? The paying dividends bit is exactly what society can't afford to cut the psychopaths at the top loose with. The only counterbalance against that rank psychopathy is the fact they'd have to have moral buy in from the people doing the work to implement, giving a de facto check on their power. Remove that element, remove the check.
        aspenmayer 1y ago
        > Remove that element, remove the check. I agree all around, but/and: Dividends will be collected by the ones who write such checks; we’ll all pay, as the institutional buy-in has already been priced in.
        aspenmayer 1y ago
        This is all a meandering way of saying something better quoted in full, so I shall do so below [sic]: > Those of us who stand outside the circle of this society’s definition of acceptable women; those of us who have been forged in the crucibles of difference — those of us who are poor, who are lesbians, who are Black, who are older — know that survival is not an academic skill. It is learning how to stand alone, unpopular and sometimes reviled, and how to make common cause with those others identified as outside the structures in order to define and seek a world in which we can all flourish. It is learning how to take our differences and make them strengths. For the master’s tools will never dismantle the master’s house. They may allow us temporarily to beat him at his own game, but they will never enable us to bring about genuine change. https://en.wikiquote.org/wiki/Audre_Lorde https://en.wikiquote.org/wiki/Audre_Lorde
- throwaway314155 1y ago
  > If your conspiratorial mind has the power to override the most obvious survival instinct, you aren't long for this world. Why even write this? It comes across as so cold. Like mentally ill people just deserve to die since they are destined to die? Incredibly reductive and toxic.
- aspenmayer 1y ago
  There’s a reason that we look forward to the Darwin Awards, but hope we and no one we know wins one. This feels like a rationalization of the bystander effect by proxy. https://en.wikipedia.org/wiki/Bystander_effect https://en.wikipedia.org/wiki/Bystander_effect

lambda 1y ago

Psychosis is not necessarily something that you are or aren't; you can be prone to it, without it having manifested, and there can be external things that trigger it. It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes. And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio. Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.

Vetch 1y ago
The non-o-series models from OpenAI and non-Opus (although I have not tried the latest, so it's possible that it too joins them) from Anthropic are cloyingly sycophantic, with every other sentence of yours containing a brilliant and fascinating insight. It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.

hluska 1y ago

The writer got overly stuck on one tweet and droned on and on about 15 words of a 45 word tweet. It was really poor research when there are other less reactionary statements she could have used. Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.

lmm 1y ago

ChatGPT writes extremely persuasively for whatever claim you ask it to make - that's pretty much what it's designed to do! If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?

bawolff 1y ago
When people blame traditional religion for doing crazy things we generally do say they were already insane.
spacechild1 1y ago
> If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"? Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.
- ath92 1y ago
  And in those cases, most people condemn it. But when it’s an algorithm (which was created by humans) we say “these people were already insane, and the creators of the algorithm should not be held liable”, apparently.
  - 9dev 1y ago
    I don’t think this analogy is fully apt. Instead, consider people going to a church or mosque. Some listen to the sermons and come out just fine, a few are radicalised over time and go find these bad people convincing them to act out on that. But drawing the line between legitimate religious teachers and ideological deceivers is hard, and condemning the entire religion would be the wrong conclusion.
    - lmm 1y ago

nxvo76 1y ago

More importantly "sane" people are fully capable of unconciously making "insane" people jump of the roof. Its not even possible to count how many times that happens. One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto. Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.

MyOutfitIsVague 1y ago

Psychosis is not a binary, and there are strong genetic components, but it's not the entire story. People develop psychoses where there weren't any before all the time, often but not necessarily in response to trauma. Some people are more predisposed to it than others. Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"? If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.

ulrikrasmussen 1y ago
> If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous. Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.
Eddy_Viscosity2 1y ago
Another danger, that I guess we just have to live with now, is not just that ChatGPT could Jim Jones people, but that someone who didn't have the natural charisma of a cult leader could use ChatGPT on purpose to Jim Jones people. It may not work so well in person, but online ... yikes.
joquarky 1y ago
Libraries also contain dangerous information.

antonvs 1y ago

> Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot. The Joe Rogan of AI.

hooverd 1y ago

It's friction. It's easier to become addicted to gambling if you can gamble from your phone vs having to go to a casino or a bookie. It's easier to become addicted to drugs or alcohol if you're awash in drugs or alcohol. Prompting an LLM can mirror the same rush as gambling or gacha games. With ChatGPT, you have a "friend" who is never going to tell you to knock it off. "Psychosis" isn't an on/off switch.

joquarky 1y ago

He's been a bit out there for a while. Remember how he reacted to Roko's Basilisk?