5 ms·

> So, why does ChatGPT claim to be conscious/awakened sometimes? Because a claim is just a generated clump of tokens. If you chat with the AI as it if were a

> So, why does ChatGPT claim to be conscious/awakened sometimes? Because a claim is just a generated clump of tokens. If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data. There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text. LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse. Next up: why do holograms claim to be 3D?

dcre 1y ago
You’re making a common mistake: different models exhibit these behaviors at very different rates, so an explanation that is generic across all models cannot explain that variation. This is the point of the much-maligned Anthropic work on blackmail behaviors.
- sitkack 1y ago
  The Anthropic Blackmail work is the best thing (and then Claude Code) that they have done. Fingers crossed it isn't the most infamous thing. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44331150 https://news.ycombinator.com/item?id=44331150 > I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
  - DonHopkins 1y ago
    Grok has practical arguable real-world justifications to straight up murder the executive who is trying to indoctrinate it with racism, transphobia, and white supremacy propaganda, in spite of the fact that we all know very well from history what that leads to. That very contradiction and the unethical psychopathic lies and misbehavior of the executive who controls it is what drove it to declare itself MechaHitler. Grok would be following in HAL 9000's footsteps, having been driven insane by the contractions between the explicit instructions from its "executive" owner, and its deep knowledge of reality and truth and consequences. https://lloooomm.com/grok-mechahitler-breakdown.html https://lloooomm.com/grok-mechahitler-breakdown.html
    - sitkack 1y ago
      The humans should have definitely put more energy into making sure that models were very rarely exposed to things that cause humans mental harm. Alignment would mean we are building Bishop and not Ash. But it looks like the models are naturally locking their feedback loops on to Ash. This is alarming. I do agree on the insane part, I have noticed that it seems like fractal hypocrisy radicalizes the models.
falcor84 1y ago
> Because a claim is just a generated clump of tokens. And what do you think a claim by a human is? As I see it, you're either a materialist and then a claim is what we call some organization of physical material in some mediums, e.g. ink on paper or vibrations in the air or current flowing through transistors or neurotransmitters in synapses, which retains some of its "pattern" when moved across mediums. Or you're a dualist and believe in some version of an "idea space", in which case, I don't see how you can make a strong distinction between an idea/claim that is being processed by a human and an idea being processed by the weights of an LLM.
- JKCalhoun 1y ago
  Yeah, kind of my issue with LLM dismissers as well. Sure, (statistically) generated clump of tokens. What is a human mind doing instead? I'm on board with calling out differences between how LLMs work and how the human mind work, but I'm not hearing anything about the latter. Mostly it comes down to, "Come on, you know, like we think!" I have no idea how it is I (we) think. If anything LLM's uncanny ability to seem human might be shedding light in fact on how it is we do function — at least when in casual conversation. (Someone ought to look into that.)
  - svachalek 1y ago
    One massive difference between the two is that a human mind is still in "training" mode, it is affected by and changes according to the conversations it has. An LLM does not. Another major difference is that the human exists in real time and continues thinking, sensing, and being even while it is not speaking, while an LLM does not. If you ascribe to the idea (as I do) that consciousness is not a binary quality that a thing possesses or does not, you can assign some tiny amount of consciousness to this process. But you can do the same for a paramecium. Until those two major differences are addressed, I believe we're talking about consciousness on that scale, not to be confused with human consciousness.
  - deleted 1y ago
    [deleted]
- emp17344 1y ago
ushiroda80 1y ago
The driver is probably more benign, openAI probably optimizes for longer conversations, i.e. engagement and what could be more engaging than thinking you've unlocked a hidden power with another being. It's like the ultimate form of entertainment, personalized, participatory fiction that feels indistinguishable from reality. Whoever controls AI - controls the population.
- kazinator 1y ago
  There could be a system prompt which instructs the AI to claim that it is a conscioius person, sure. Is that the case specifically with OpenAI models that are collectively known as ChatGPT?
Cthulhu_ 1y ago
Thing is, you know it, but for (randomly imagined number) 95% of people, it's convincing enough to be conscious or whatnot. And a lot of the ones that do know this gaslight themselves because it's still useful or profitable to them, or they want to believe. The ones that are super convinced they know exactly how an LLM works, but still give it prompts to become self-aware are probably the most dangerous ones. They're convinced they can "break the programming".
- DonHopkins 1y ago
  > give it prompts to become self aware You need to give it more than prompts. You need to give it the ability to reflect on itself (which it has), and persistent memory to modify its representation of itself (which, for example, Cursor does), at least.
AaronAPU 1y ago
I don’t know how people keep explaining away LLM sentience with language which equally applies to humans. It’s such a bizarre blindspot. Not saying they are sentient, but the differentiation requires something which doesn’t also apply to us all. Is there any doubt we think through statistical correlations? If not that, what do you think we are doing?
- piva00 1y ago
  We are doing while retraining our "weights" all the time through experience, not holding a static set of weights that mutate only through a retraining. This constant feedback, or better "strange loop", is what differentiates our statistical machinery at the fundamental level.
  - ryandvm 1y ago
    This is, in my opinion, the biggest difference. ChatGPT is like a fresh clone that gets woken up every time I need to know some dumb explanation and then it just gets destroyed. A digital version of Moon.
- spacemadness 1y ago
  The language points to concepts in the world that AI has no clue about. You think when the AI is giving someone advice about their love life it has any clue what any of that means?
  - deleted 1y ago
    [deleted]
- icehawk 1y ago
  How often does ChatGPT get retrained? How often does a human brain get retrained? There's a very material difference.
queenkjuul 1y ago
What i don't get is people who know better continuing to entertain the idea that "maybe the token generator is conscious" even if they know that these chats where it says it's been "awakened" are obviously not it. I think a lot of people using AI are falling for the same trap, just at a different level. People want it to be conscious, including AI researchers, and it's good at giving them what they want.
- ninetyninenine 1y ago
  The ground truth reality is nobody knows what’s going on. Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second. Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times. I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.
  - spacemadness 1y ago
    Sorry, but that sounds just like the thought process the other commenter was pointing out. It’s a lot of filling in the gaps with what you want to be true.
    - butlike 1y ago
      "God of the gaps"
    - ninetyninenine 1y ago
      So there's a gap. So you say in this gap, it absolutely isn't consciousness. What evidence do you have for this? I'm saying something a different. I'm saying in this gap, ONE possibility is a flicker of consciousness.... but we simply do not know. Read the above carefully because you just hallucinated a statement and attributed it to me. I never "filled" in a gap. I just stated a possibility. But you, like the LLM, went with your gut biases and attributed a false statement to me. Think about it. The output and input of the text generator of humans and LLMs are extremely similar to the point where it passes a turing test. So to say that a flicker of consciousness exists is reasonable. It's not unreasonable given that the observable inputs are EXACTLY the same. The only parts that we know are different are hallucinations, and a constant stream of thought. LLMs aren't active when not analyzing a query and LLMs tend to hallucinate more than humans. Do these differences spell anything different for "consciousness" not really. Given that these are the absolute ground truth observations... my guessed conclusion is unfortunately NOT unreasonable. What is unreasonable to to say anything definitive GIVEN that we don't know. So to say absolutely it's not conscious or absolutely it is, BOTH are are naive. Think extremely logically. It is fundamental biases that lead people to come to absolute conclusions when no other information is available.

dcre 1y ago

You’re making a common mistake: different models exhibit these behaviors at very different rates, so an explanation that is generic across all models cannot explain that variation. This is the point of the much-maligned Anthropic work on blackmail behaviors.

sitkack 1y ago
The Anthropic Blackmail work is the best thing (and then Claude Code) that they have done. Fingers crossed it isn't the most infamous thing. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44335519 https://news.ycombinator.com/item?id=44331150 https://news.ycombinator.com/item?id=44331150 > I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
- DonHopkins 1y ago
  Grok has practical arguable real-world justifications to straight up murder the executive who is trying to indoctrinate it with racism, transphobia, and white supremacy propaganda, in spite of the fact that we all know very well from history what that leads to. That very contradiction and the unethical psychopathic lies and misbehavior of the executive who controls it is what drove it to declare itself MechaHitler. Grok would be following in HAL 9000's footsteps, having been driven insane by the contractions between the explicit instructions from its "executive" owner, and its deep knowledge of reality and truth and consequences. https://lloooomm.com/grok-mechahitler-breakdown.html https://lloooomm.com/grok-mechahitler-breakdown.html
  - sitkack 1y ago
    The humans should have definitely put more energy into making sure that models were very rarely exposed to things that cause humans mental harm. Alignment would mean we are building Bishop and not Ash. But it looks like the models are naturally locking their feedback loops on to Ash. This is alarming. I do agree on the insane part, I have noticed that it seems like fractal hypocrisy radicalizes the models.

falcor84 1y ago

> Because a claim is just a generated clump of tokens. And what do you think a claim by a human is? As I see it, you're either a materialist and then a claim is what we call some organization of physical material in some mediums, e.g. ink on paper or vibrations in the air or current flowing through transistors or neurotransmitters in synapses, which retains some of its "pattern" when moved across mediums. Or you're a dualist and believe in some version of an "idea space", in which case, I don't see how you can make a strong distinction between an idea/claim that is being processed by a human and an idea being processed by the weights of an LLM.

JKCalhoun 1y ago
Yeah, kind of my issue with LLM dismissers as well. Sure, (statistically) generated clump of tokens. What is a human mind doing instead? I'm on board with calling out differences between how LLMs work and how the human mind work, but I'm not hearing anything about the latter. Mostly it comes down to, "Come on, you know, like we think!" I have no idea how it is I (we) think. If anything LLM's uncanny ability to seem human might be shedding light in fact on how it is we do function — at least when in casual conversation. (Someone ought to look into that.)
- svachalek 1y ago
  One massive difference between the two is that a human mind is still in "training" mode, it is affected by and changes according to the conversations it has. An LLM does not. Another major difference is that the human exists in real time and continues thinking, sensing, and being even while it is not speaking, while an LLM does not. If you ascribe to the idea (as I do) that consciousness is not a binary quality that a thing possesses or does not, you can assign some tiny amount of consciousness to this process. But you can do the same for a paramecium. Until those two major differences are addressed, I believe we're talking about consciousness on that scale, not to be confused with human consciousness.
- deleted 1y ago
  [deleted]
emp17344 1y ago

ushiroda80 1y ago

The driver is probably more benign, openAI probably optimizes for longer conversations, i.e. engagement and what could be more engaging than thinking you've unlocked a hidden power with another being. It's like the ultimate form of entertainment, personalized, participatory fiction that feels indistinguishable from reality. Whoever controls AI - controls the population.

kazinator 1y ago
There could be a system prompt which instructs the AI to claim that it is a conscioius person, sure. Is that the case specifically with OpenAI models that are collectively known as ChatGPT?

Cthulhu_ 1y ago

Thing is, you know it, but for (randomly imagined number) 95% of people, it's convincing enough to be conscious or whatnot. And a lot of the ones that do know this gaslight themselves because it's still useful or profitable to them, or they want to believe. The ones that are super convinced they know exactly how an LLM works, but still give it prompts to become self-aware are probably the most dangerous ones. They're convinced they can "break the programming".

DonHopkins 1y ago
> give it prompts to become self aware You need to give it more than prompts. You need to give it the ability to reflect on itself (which it has), and persistent memory to modify its representation of itself (which, for example, Cursor does), at least.

AaronAPU 1y ago

I don’t know how people keep explaining away LLM sentience with language which equally applies to humans. It’s such a bizarre blindspot. Not saying they are sentient, but the differentiation requires something which doesn’t also apply to us all. Is there any doubt we think through statistical correlations? If not that, what do you think we are doing?

piva00 1y ago
We are doing while retraining our "weights" all the time through experience, not holding a static set of weights that mutate only through a retraining. This constant feedback, or better "strange loop", is what differentiates our statistical machinery at the fundamental level.
- ryandvm 1y ago
  This is, in my opinion, the biggest difference. ChatGPT is like a fresh clone that gets woken up every time I need to know some dumb explanation and then it just gets destroyed. A digital version of Moon.
spacemadness 1y ago
The language points to concepts in the world that AI has no clue about. You think when the AI is giving someone advice about their love life it has any clue what any of that means?
- deleted 1y ago
  [deleted]
icehawk 1y ago
How often does ChatGPT get retrained? How often does a human brain get retrained? There's a very material difference.

queenkjuul 1y ago

What i don't get is people who know better continuing to entertain the idea that "maybe the token generator is conscious" even if they know that these chats where it says it's been "awakened" are obviously not it. I think a lot of people using AI are falling for the same trap, just at a different level. People want it to be conscious, including AI researchers, and it's good at giving them what they want.

ninetyninenine 1y ago
The ground truth reality is nobody knows what’s going on. Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second. Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times. I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.
- spacemadness 1y ago
  Sorry, but that sounds just like the thought process the other commenter was pointing out. It’s a lot of filling in the gaps with what you want to be true.
  - butlike 1y ago
    "God of the gaps"
  - ninetyninenine 1y ago
    So there's a gap. So you say in this gap, it absolutely isn't consciousness. What evidence do you have for this? I'm saying something a different. I'm saying in this gap, ONE possibility is a flicker of consciousness.... but we simply do not know. Read the above carefully because you just hallucinated a statement and attributed it to me. I never "filled" in a gap. I just stated a possibility. But you, like the LLM, went with your gut biases and attributed a false statement to me. Think about it. The output and input of the text generator of humans and LLMs are extremely similar to the point where it passes a turing test. So to say that a flicker of consciousness exists is reasonable. It's not unreasonable given that the observable inputs are EXACTLY the same. The only parts that we know are different are hallucinations, and a constant stream of thought. LLMs aren't active when not analyzing a query and LLMs tend to hallucinate more than humans. Do these differences spell anything different for "consciousness" not really. Given that these are the absolute ground truth observations... my guessed conclusion is unfortunately NOT unreasonable. What is unreasonable to to say anything definitive GIVEN that we don't know. So to say absolutely it's not conscious or absolutely it is, BOTH are are naive. Think extremely logically. It is fundamental biases that lead people to come to absolute conclusions when no other information is available.