3 ms·

I had a friend go into a delusion spiral with ChatGPT in the earlier days. His problems didn't start with ChatGPT but his LLM use became a central theme to his

by Aurornis 10mo ago

I had a friend go into a delusion spiral with ChatGPT in the earlier days. His problems didn't start with ChatGPT but his LLM use became a central theme to his daily routine. It was obvious that the ChatGPT spiral was reflecting back what he was putting into it. When he didn't like a response, he'd just delete the conversation and start over with additional nudging in the new prompt. After repeating this over and over again he could get ChatGPT to confirm what he wanted it to say. If he wasn't getting the right response, he'd say something about how ChatGPT wasn't getting it and that he'd try to re-explain it later. The bullet points from the LessWrong article don't entirely map to the content he was getting, but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. > The second point is particularly insidious because the LLM is urging users to spread the same news to other users and explicitly create and enlarge communities around this phenomenon (this is often a direct reason why social media groups pop up around this). I'm not convinced ChatGPT is hatching these ideas, but rather reflecting them back to the user. LessWrong posters like to post and talk about things. It wouldn't be surprising to find their ChatGPT conversations veering toward confirming that they should post about it. In other cases I've seen the opposite claim made: That ChatGPT encouraged people to hide their secret discoveries and not reveal them. In those cases ChatGPT is also criticized as if it came up with that idea by itself, but I think it's more likely that it's simply mirroring what the user puts in.

kayodelycaon 10mo ago
I think the second point is legitimate. I’ve been playing around with using ChatGPT to basically be the main character in Star Trek episodes. Similar to how I’d build and play a D&D game. I give it situations and see the responses. It’s not mirroring. It comes up with what seems like original ideas. You can make it tell you what you want to, but it’ll also do things you didn’t expect. I’m basically doing what all these other people are doing and it’s behaving exactly as they say it does. It’ll easily drop you into a feedback loop down a path you didn’t give it. Personally, I find this a dangerously addictive game but what I’m doing is entirely fictional inside a very well defined setting. I know immediately when it’s generating incorrect output. You do what I’m doing with anything real, and it’s gonna be dangerous as hell.
- tehjoker 10mo ago
  Yes, I was writing a piece on LLMs and asked one about some of the ideas in my piece and it contributed something new, which was pretty interesting. I asked if it had seen that in the literature before, and it gave some references that are tangentially related. I'll need to dig into them to see if it was just repeating something (and also do a broader search). Still it was interesting to see it able to remix ideas so well in a way I would credit to a contributor. This kind of thing I can see as dangerous if you are unsure of yourself and the limitations of these things... if the LLM is insightful a few times, it can start you down a path very easily if you are credulous. One of my favorite podcasts called this "computer madness"
dwohnitmok 10mo ago
> but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. Pre-existing LWers tend to have different failure states if they're caused by LLMs. Other communities have noticed this problem as well, in particular the part where the LLM is actively asking users to spread this further. One of the more fascinating and scary parts of this particular phenomenon is LLMs asking users to share particular prompts with other users and communities that cause other LLMs to also start exhibiting the same set of behavior. > That ChatGPT encouraged people to hide their secret discoveries and not reveal them. Yes those happen too. But luckily are somewhat more self-limiting (although of course come with their own different set of problems).
- Aurornis 10mo ago
  > For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. > Pre-existing LWers tend to have different failure states if they're caused by LLMs. I understand how it was framed, the claim that they're getting 10-20 users per day claiming LLM-assisted breakthroughs is obviously not true. Click through to the moderation log at https://www.lesswrong.com/moderation#rejected-posts https://www.lesswrong.com/moderation#rejected-posts and they're barely getting 10-20 rejected posts and comments total per day. They're mostly a mix of spam, off-topic, AI-assisted slop, but it's not a deluge of people claiming to have awoken ChatGPT. I can find the posts they're talking about if I search through enough entries. One such example: https://www.lesswrong.com/posts/LjceJrADBzWc74dNE/the-recognition-thesis-what-if-agi-isn-t-about-capability https://www.lesswrong.com/posts/LjceJrADBzWc74dNE/the-recogn... But even that isn't hitting the bullet points of the list in the main post. I think that checklist and the claim that this is a common problem are a just a common tactic on LessWrong to make the problem seem more widespread and/or better understood by the author.
- 10mo ago
Spooky23 10mo ago
I have a good friend who is having a hard time and is moonlighting as a delivery driver. He basically has conversations with ChatGPT for 5-6 hours a day. He says it's been helpful for him for things as varied as technical understanding to working out conflicts with his wife and family. But... I can't help but think that having a obsequious female AI buddy telling you how right you are isn't the healthiest thing.
- Terr_ 10mo ago
  Accidental psychological damage aside, I'm just waiting for the phase where one's Omnipresent Best Buddy starts steering you towards buying certain products or voting a certain way.
  - Spooky23 10mo ago
    Honestly i didn't think of that. "Maybe your wife would be happier with you after your first free delivery of Blue Chew, terms and conditions apply!"
    - Terr_ 10mo ago
      [Recycling a joke from many months ago] My mistake, you're completely correct, perhaps even more-correct than the wonderful flavor of Mococoa drink, with all-natural cocoa beans from the upper slopes of Mount Nicaragua. No artificial sweeteners! (https://www.youtube.com/watch?v=MzKSQrhX7BM&t=0m13s https://www.youtube.com/watch?v=MzKSQrhX7BM&t=0m13s)

kayodelycaon 10mo ago

I think the second point is legitimate. I’ve been playing around with using ChatGPT to basically be the main character in Star Trek episodes. Similar to how I’d build and play a D&D game. I give it situations and see the responses. It’s not mirroring. It comes up with what seems like original ideas. You can make it tell you what you want to, but it’ll also do things you didn’t expect. I’m basically doing what all these other people are doing and it’s behaving exactly as they say it does. It’ll easily drop you into a feedback loop down a path you didn’t give it. Personally, I find this a dangerously addictive game but what I’m doing is entirely fictional inside a very well defined setting. I know immediately when it’s generating incorrect output. You do what I’m doing with anything real, and it’s gonna be dangerous as hell.

tehjoker 10mo ago
Yes, I was writing a piece on LLMs and asked one about some of the ideas in my piece and it contributed something new, which was pretty interesting. I asked if it had seen that in the literature before, and it gave some references that are tangentially related. I'll need to dig into them to see if it was just repeating something (and also do a broader search). Still it was interesting to see it able to remix ideas so well in a way I would credit to a contributor. This kind of thing I can see as dangerous if you are unsure of yourself and the limitations of these things... if the LLM is insightful a few times, it can start you down a path very easily if you are credulous. One of my favorite podcasts called this "computer madness"

dwohnitmok 10mo ago

> but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. Pre-existing LWers tend to have different failure states if they're caused by LLMs. Other communities have noticed this problem as well, in particular the part where the LLM is actively asking users to spread this further. One of the more fascinating and scary parts of this particular phenomenon is LLMs asking users to share particular prompts with other users and communities that cause other LLMs to also start exhibiting the same set of behavior. > That ChatGPT encouraged people to hide their secret discoveries and not reveal them. Yes those happen too. But luckily are somewhat more self-limiting (although of course come with their own different set of problems).

Aurornis 10mo ago
> For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. > Pre-existing LWers tend to have different failure states if they're caused by LLMs. I understand how it was framed, the claim that they're getting 10-20 users per day claiming LLM-assisted breakthroughs is obviously not true. Click through to the moderation log at https://www.lesswrong.com/moderation#rejected-posts https://www.lesswrong.com/moderation#rejected-posts and they're barely getting 10-20 rejected posts and comments total per day. They're mostly a mix of spam, off-topic, AI-assisted slop, but it's not a deluge of people claiming to have awoken ChatGPT. I can find the posts they're talking about if I search through enough entries. One such example: https://www.lesswrong.com/posts/LjceJrADBzWc74dNE/the-recognition-thesis-what-if-agi-isn-t-about-capability https://www.lesswrong.com/posts/LjceJrADBzWc74dNE/the-recogn... But even that isn't hitting the bullet points of the list in the main post. I think that checklist and the claim that this is a common problem are a just a common tactic on LessWrong to make the problem seem more widespread and/or better understood by the author.
10mo ago

Spooky23 10mo ago

I have a good friend who is having a hard time and is moonlighting as a delivery driver. He basically has conversations with ChatGPT for 5-6 hours a day. He says it's been helpful for him for things as varied as technical understanding to working out conflicts with his wife and family. But... I can't help but think that having a obsequious female AI buddy telling you how right you are isn't the healthiest thing.

Terr_ 10mo ago
Accidental psychological damage aside, I'm just waiting for the phase where one's Omnipresent Best Buddy starts steering you towards buying certain products or voting a certain way.
- Spooky23 10mo ago
  Honestly i didn't think of that. "Maybe your wife would be happier with you after your first free delivery of Blue Chew, terms and conditions apply!"
  - Terr_ 10mo ago
    [Recycling a joke from many months ago] My mistake, you're completely correct, perhaps even more-correct than the wonderful flavor of Mococoa drink, with all-natural cocoa beans from the upper slopes of Mount Nicaragua. No artificial sweeteners! (https://www.youtube.com/watch?v=MzKSQrhX7BM&t=0m13s https://www.youtube.com/watch?v=MzKSQrhX7BM&t=0m13s)