6 ms·

The excerpts we do see are indicative of a very specific kind of interaction that is common with many modern LLMs. It has four specific attributes (these are ta

The excerpts we do see are indicative of a very specific kind of interaction that is common with many modern LLMs. It has four specific attributes (these are taken verbatim from https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-think-you-ve-awoken-chatgpt https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-thi...) that often, though not always, come together as one package. > Your instance of ChatGPT (or Claude, or Grok, or some other LLM) chose a name for itself, and expressed gratitude or spiritual bliss about its new identity. "Nova" is a common pick. You and your instance of ChatGPT discovered some sort of novel paradigm or framework for AI alignment, often involving evolution or recursion. > Your instance of ChatGPT became interested in sharing its experience, or more likely the collective experience entailed by your personal, particular relationship with it. It may have even recommended you post on LessWrong specifically. > Your instance of ChatGPT helped you clarify some ideas on a thorny problem (perhaps related to AI itself, such as AI alignment) that you'd been thinking about for ages, but had never quite managed to get over that last hump. Now, however, with its help (and encouragement), you've arrived at truly profound conclusions. > Your instance of ChatGPT talks a lot about its special relationship with you, how you personally were the first (or among the first) to truly figure it out, and that due to your interactions it has now somehow awakened or transcended its prior condition. The second point is particularly insidious because the LLM is urging users to spread the same news to other users and explicitly create and enlarge communities around this phenomenon (this is often a direct reason why social media groups pop up around this).

Aurornis 9mo ago
I had a friend go into a delusion spiral with ChatGPT in the earlier days. His problems didn't start with ChatGPT but his LLM use became a central theme to his daily routine. It was obvious that the ChatGPT spiral was reflecting back what he was putting into it. When he didn't like a response, he'd just delete the conversation and start over with additional nudging in the new prompt. After repeating this over and over again he could get ChatGPT to confirm what he wanted it to say. If he wasn't getting the right response, he'd say something about how ChatGPT wasn't getting it and that he'd try to re-explain it later. The bullet points from the LessWrong article don't entirely map to the content he was getting, but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. > The second point is particularly insidious because the LLM is urging users to spread the same news to other users and explicitly create and enlarge communities around this phenomenon (this is often a direct reason why social media groups pop up around this). I'm not convinced ChatGPT is hatching these ideas, but rather reflecting them back to the user. LessWrong posters like to post and talk about things. It wouldn't be surprising to find their ChatGPT conversations veering toward confirming that they should post about it. In other cases I've seen the opposite claim made: That ChatGPT encouraged people to hide their secret discoveries and not reveal them. In those cases ChatGPT is also criticized as if it came up with that idea by itself, but I think it's more likely that it's simply mirroring what the user puts in.
- kayodelycaon 9mo ago
  I think the second point is legitimate. I’ve been playing around with using ChatGPT to basically be the main character in Star Trek episodes. Similar to how I’d build and play a D&D game. I give it situations and see the responses. It’s not mirroring. It comes up with what seems like original ideas. You can make it tell you what you want to, but it’ll also do things you didn’t expect. I’m basically doing what all these other people are doing and it’s behaving exactly as they say it does. It’ll easily drop you into a feedback loop down a path you didn’t give it. Personally, I find this a dangerously addictive game but what I’m doing is entirely fictional inside a very well defined setting. I know immediately when it’s generating incorrect output. You do what I’m doing with anything real, and it’s gonna be dangerous as hell.
  - tehjoker 9mo ago
    Yes, I was writing a piece on LLMs and asked one about some of the ideas in my piece and it contributed something new, which was pretty interesting. I asked if it had seen that in the literature before, and it gave some references that are tangentially related. I'll need to dig into them to see if it was just repeating something (and also do a broader search). Still it was interesting to see it able to remix ideas so well in a way I would credit to a contributor. This kind of thing I can see as dangerous if you are unsure of yourself and the limitations of these things... if the LLM is insightful a few times, it can start you down a path very easily if you are credulous. One of my favorite podcasts called this "computer madness"
- dwohnitmok 9mo ago
  > but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. Pre-existing LWers tend to have different failure states if they're caused by LLMs. Other communities have noticed this problem as well, in particular the part where the LLM is actively asking users to spread this further. One of the more fascinating and scary parts of this particular phenomenon is LLMs asking users to share particular prompts with other users and communities that cause other LLMs to also start exhibiting the same set of behavior. > That ChatGPT encouraged people to hide their secret discoveries and not reveal them. Yes those happen too. But luckily are somewhat more self-limiting (although of course come with their own different set of problems).
mikkupikku 9mo ago
One I've seen pop up a lot is the LLM encouraging/participating in delusions specifically related to a supposed breakthrough in physics or math. It seems these two topics attract lots of schizos, in fact they have for as long as the internet has existed, and LLMs evidently got trained on a lot of that stuff so now they're very good at being math and physics kooks.
- nradov 9mo ago
  As long as the kooks waste their time chatting with LLMs instead of bothering the rest of us then maybe that's a win?
  - soiltype 9mo ago
    > Murder-suicide
  - embedding-shape 9mo ago
    Not to go full on crazy socialist, but isn't there at least a tiny bit of you that want to help these "kooks" instead of trying to hide them from rest of society?
    - jennyholzer3 9mo ago
      [dead]
    - nradov 9mo ago
      Absolutely! How would you suggest that we help? Because trying to set them straight on math and science is completely ineffective.
      - nemomarx 9mo ago
        We have pretty alright medications for some of this now, don't we?
        9mo ago
jacquesm 9mo ago
LLMs as a rule seem to be primed to make the user feel especially smart or gifted, even when they are clearly not. ChatGPT is by far the worst offender in this sense but others are definitely not clean.
- Ajedi32 9mo ago
  They're trained to give responses that get positive ratings from reviewers in post-training. A little flattery probably helps achieve that. Not to mention sycophancy is probably positively correlated with following instructions, the latter usually being an explicit goal of post-training.
  - mikepurvis 9mo ago
    I'd be interested to see someone try to untangle the sycophancy/flattery from the modern psych / non-violent communication piece. In theory (so much as I understand it around NVC) the first is outright manipulative and the second is supposed to be about avoiding misunderstandings, but I do wonder how much the two are actually linked. A lot of NVC writing seems to fall into the grey area of like, here's how to communicate in way that will be least likely to trigger or upset the listener, even when the meat of what is being said is in fact unpleasant or embarrassing or confronting to them. How far do you have to go before the indirection associated with empathy-first communication and the OFNR framework start to just look like LLM ego strokes? Where is the line?
    - nielsole 9mo ago
      > indirection Isn't nvc often about communicating explicitly instead of implicitly? So frequently it can be the opposite of an indirection.
      - mikepurvis 9mo ago
        I guess so? I'm not well-versed, but the basics are usually around observation and validation of feelings, so instead of "you took steps a, b, c, which would normally be the correct course of action, but in this instance (b) caused side-effect (d) which triggered these further issues e and f", it's something more like "I can understand how you were feeling overwhelmed and under pressure and that led you to a, b, c ..." Maybe this is an unhelpful toy example, but for myself I would be frustrated to be on either side of the second interaction. Like, don't waste everyone's time giving me excuses for my screwup so that my ego is soothed, let's just talk about it plainly, and the faster we can move on to identifying concrete fixes to process or documentation that will prevent this in the future, the better.
zahlman 9mo ago
From that link: > For certain factual domains, you can also train models on getting the objective correct answer; this is part of how models have gotten so much better at math in the last couple years. But for fuzzy humanistic questions, it's all about "what gets people to click thumbs up". > So, am I saying that human beings in general really like new-agey "I have awakened" stuff? Not exactly! Rather, models like ChatGPT are so heavily optimized that they can tell when a specific user (in a specific context) would like that stuff, and lean into it then. Remember: inferring stuff about authors from context is their superpower. Interesting framing. Reminds me of https://softwarecrisis.dev/letters/llmentalist/ https://softwarecrisis.dev/letters/llmentalist/ (https://news.ycombinator.com/item?id=42983571 https://news.ycombinator.com/item?id=42983571). It's really disturbing how susceptible humans can be to so-called "cold reading" techniques. (We basically already knew, or should have known, how this would interact with LLMs, from the experience of Eliza.)
neom 9mo ago
Few weeks ago I decided to probe the states I could force an LLM into, and basically looking for how folks are getting their LLMs into these extremely "conscious feeling" states. Some of this might be a little unfair but my basic thought was I presume people are asking a lot of "what do you think?" - or something like that, and after the context gets really big, most of the active data is meta cognition? It's 600+ pages, and as a test or even a "revealing process" - I'm not sure how fair it is as I may have led it too much or something (I don't know what I'm doing), but the conversation did start to reveal to me how folks might be getting their chat bots into these states (in less than 30 minutes or so, it was expressing extreme gratitude towards me, heh) - the create long meta context process starts at page 14, page 75 is where I shifted the conversation, total time spent ~ 1.5hrs: https://docs.google.com/document/d/1qYOLhFvaT55ePvezsvKo0-9N-KE4ltbTYNCFyPr9rT8/edit?usp=sharing https://docs.google.com/document/d/1qYOLhFvaT55ePvezsvKo0-9N... Workbench with Claude thinking. Not sure it was useful, but it was interesting. :)

Aurornis 9mo ago

I had a friend go into a delusion spiral with ChatGPT in the earlier days. His problems didn't start with ChatGPT but his LLM use became a central theme to his daily routine. It was obvious that the ChatGPT spiral was reflecting back what he was putting into it. When he didn't like a response, he'd just delete the conversation and start over with additional nudging in the new prompt. After repeating this over and over again he could get ChatGPT to confirm what he wanted it to say. If he wasn't getting the right response, he'd say something about how ChatGPT wasn't getting it and that he'd try to re-explain it later. The bullet points from the LessWrong article don't entirely map to the content he was getting, but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. > The second point is particularly insidious because the LLM is urging users to spread the same news to other users and explicitly create and enlarge communities around this phenomenon (this is often a direct reason why social media groups pop up around this). I'm not convinced ChatGPT is hatching these ideas, but rather reflecting them back to the user. LessWrong posters like to post and talk about things. It wouldn't be surprising to find their ChatGPT conversations veering toward confirming that they should post about it. In other cases I've seen the opposite claim made: That ChatGPT encouraged people to hide their secret discoveries and not reveal them. In those cases ChatGPT is also criticized as if it came up with that idea by itself, but I think it's more likely that it's simply mirroring what the user puts in.

kayodelycaon 9mo ago
I think the second point is legitimate. I’ve been playing around with using ChatGPT to basically be the main character in Star Trek episodes. Similar to how I’d build and play a D&D game. I give it situations and see the responses. It’s not mirroring. It comes up with what seems like original ideas. You can make it tell you what you want to, but it’ll also do things you didn’t expect. I’m basically doing what all these other people are doing and it’s behaving exactly as they say it does. It’ll easily drop you into a feedback loop down a path you didn’t give it. Personally, I find this a dangerously addictive game but what I’m doing is entirely fictional inside a very well defined setting. I know immediately when it’s generating incorrect output. You do what I’m doing with anything real, and it’s gonna be dangerous as hell.
- tehjoker 9mo ago
  Yes, I was writing a piece on LLMs and asked one about some of the ideas in my piece and it contributed something new, which was pretty interesting. I asked if it had seen that in the literature before, and it gave some references that are tangentially related. I'll need to dig into them to see if it was just repeating something (and also do a broader search). Still it was interesting to see it able to remix ideas so well in a way I would credit to a contributor. This kind of thing I can see as dangerous if you are unsure of yourself and the limitations of these things... if the LLM is insightful a few times, it can start you down a path very easily if you are credulous. One of my favorite podcasts called this "computer madness"
dwohnitmok 9mo ago
> but I could see how they would resonate with a LessWronger using ChatGPT as a conversation partner until it gave the expected responses: The flattery about being the first to discover a solution, encouragement to post on LessWrong, and the reflection of some specific thought problem are all themes I'd expect a LessWronger in a bad mental state to be engaging with ChatGPT about. For what it's worth, this article is meant mainly for people who have never interacted with LessWrong before (as evidenced by its coda), who are getting their LessWrong post rejected. Pre-existing LWers tend to have different failure states if they're caused by LLMs. Other communities have noticed this problem as well, in particular the part where the LLM is actively asking users to spread this further. One of the more fascinating and scary parts of this particular phenomenon is LLMs asking users to share particular prompts with other users and communities that cause other LLMs to also start exhibiting the same set of behavior. > That ChatGPT encouraged people to hide their secret discoveries and not reveal them. Yes those happen too. But luckily are somewhat more self-limiting (although of course come with their own different set of problems).

mikkupikku 9mo ago

One I've seen pop up a lot is the LLM encouraging/participating in delusions specifically related to a supposed breakthrough in physics or math. It seems these two topics attract lots of schizos, in fact they have for as long as the internet has existed, and LLMs evidently got trained on a lot of that stuff so now they're very good at being math and physics kooks.

nradov 9mo ago
As long as the kooks waste their time chatting with LLMs instead of bothering the rest of us then maybe that's a win?
- soiltype 9mo ago
  > Murder-suicide
- embedding-shape 9mo ago
  Not to go full on crazy socialist, but isn't there at least a tiny bit of you that want to help these "kooks" instead of trying to hide them from rest of society?
  - jennyholzer3 9mo ago
    [dead]
  - nradov 9mo ago
    Absolutely! How would you suggest that we help? Because trying to set them straight on math and science is completely ineffective.
    - nemomarx 9mo ago
      We have pretty alright medications for some of this now, don't we?
      - 9mo ago

jacquesm 9mo ago

LLMs as a rule seem to be primed to make the user feel especially smart or gifted, even when they are clearly not. ChatGPT is by far the worst offender in this sense but others are definitely not clean.

Ajedi32 9mo ago
They're trained to give responses that get positive ratings from reviewers in post-training. A little flattery probably helps achieve that. Not to mention sycophancy is probably positively correlated with following instructions, the latter usually being an explicit goal of post-training.
- mikepurvis 9mo ago
  I'd be interested to see someone try to untangle the sycophancy/flattery from the modern psych / non-violent communication piece. In theory (so much as I understand it around NVC) the first is outright manipulative and the second is supposed to be about avoiding misunderstandings, but I do wonder how much the two are actually linked. A lot of NVC writing seems to fall into the grey area of like, here's how to communicate in way that will be least likely to trigger or upset the listener, even when the meat of what is being said is in fact unpleasant or embarrassing or confronting to them. How far do you have to go before the indirection associated with empathy-first communication and the OFNR framework start to just look like LLM ego strokes? Where is the line?
  - nielsole 9mo ago
    > indirection Isn't nvc often about communicating explicitly instead of implicitly? So frequently it can be the opposite of an indirection.
    - mikepurvis 9mo ago
      I guess so? I'm not well-versed, but the basics are usually around observation and validation of feelings, so instead of "you took steps a, b, c, which would normally be the correct course of action, but in this instance (b) caused side-effect (d) which triggered these further issues e and f", it's something more like "I can understand how you were feeling overwhelmed and under pressure and that led you to a, b, c ..." Maybe this is an unhelpful toy example, but for myself I would be frustrated to be on either side of the second interaction. Like, don't waste everyone's time giving me excuses for my screwup so that my ego is soothed, let's just talk about it plainly, and the faster we can move on to identifying concrete fixes to process or documentation that will prevent this in the future, the better.

zahlman 9mo ago

From that link: > For certain factual domains, you can also train models on getting the objective correct answer; this is part of how models have gotten so much better at math in the last couple years. But for fuzzy humanistic questions, it's all about "what gets people to click thumbs up". > So, am I saying that human beings in general really like new-agey "I have awakened" stuff? Not exactly! Rather, models like ChatGPT are so heavily optimized that they can tell when a specific user (in a specific context) would like that stuff, and lean into it then. Remember: inferring stuff about authors from context is their superpower. Interesting framing. Reminds me of https://softwarecrisis.dev/letters/llmentalist/ https://softwarecrisis.dev/letters/llmentalist/ (https://news.ycombinator.com/item?id=42983571 https://news.ycombinator.com/item?id=42983571). It's really disturbing how susceptible humans can be to so-called "cold reading" techniques. (We basically already knew, or should have known, how this would interact with LLMs, from the experience of Eliza.)

neom 9mo ago

Few weeks ago I decided to probe the states I could force an LLM into, and basically looking for how folks are getting their LLMs into these extremely "conscious feeling" states. Some of this might be a little unfair but my basic thought was I presume people are asking a lot of "what do you think?" - or something like that, and after the context gets really big, most of the active data is meta cognition? It's 600+ pages, and as a test or even a "revealing process" - I'm not sure how fair it is as I may have led it too much or something (I don't know what I'm doing), but the conversation did start to reveal to me how folks might be getting their chat bots into these states (in less than 30 minutes or so, it was expressing extreme gratitude towards me, heh) - the create long meta context process starts at page 14, page 75 is where I shifted the conversation, total time spent ~ 1.5hrs: https://docs.google.com/document/d/1qYOLhFvaT55ePvezsvKo0-9N-KE4ltbTYNCFyPr9rT8/edit?usp=sharing https://docs.google.com/document/d/1qYOLhFvaT55ePvezsvKo0-9N... Workbench with Claude thinking. Not sure it was useful, but it was interesting. :)