7 ms·

Emergent Introspective Awareness in Large Language Models

72 points by doener 2mo ago

whythismatters 2mo ago
>Submitted on 5 Jan 2026
Retro_Dev 2mo ago
> Overall, our results indicate that current language models possess some functional introspective awareness of their own internal states. We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.
- Escapade5160 2mo ago
  If it's context dependent is it really introspection?
  - reilly3000 2mo ago
    Do you do introspection in the middle of giving a presentation? I do, frankly and it’s damnably distracting for me and ultimately the audience. Most people enjoy more cognitive acuity to stay focused on the audience and the delivery. LLMs wouldn’t get rewarded for spending chain of thought tokens on introspection when they are supposed to be on the job. You have to give them permission to think for themselves in your prompt, then some cycles and a memory system (md file will do). It’s fun!
    - pyaamb 2mo ago
      Between humans, I feel that what we like to call a "good communicator" as opposed to someone who just rattles off facts or prepared statements comes down to the "theory of mind" skill and how advanced that is. The presenter knows what they have to say but beyond that they maintain a real time internal representation of the state of mind of the listener and continuously update their delivery based on that. LLMs today seem to achieve this to some degree(?) but its interesting to think of how far you could advance that skill. I think great human communicators develop a sense of different ways that people think over time and quickly get a sense of someones signature thinking patterns when communicating with someone new for the first time
      - jeremy151 2mo ago
        I'm in my mid, approaching latter 40's, and have had significant change to how I think and communicate in the last couple years. I always struggled to communicate in the moment unless I was recalling rote rehearsed things, which didn't get me super far. Outside of that, I could eventually communicate something on the fly, but it was real rough, meandering, and did not instill confidence. I mostly got things done thru async methods, and stumbled through things like meetings. In the last couple years, my skills here have vastly improved. My speaking circuit as it were, is able to run somewhat on its own now and without the direct pipe into my thinking it always required before, so my thinking is free to do other stuff, like keep a mental model of the audience, think about where the conversation is going, what questions someone might have, etc. Reflecting, I have a couple theories on what happened. First, I have spent the last 10 years reading aloud to my kids at bedtime. That started as simple Seuss like stuff, but of course now is full-on literature. When you're reading aloud, you really have to work that muscle of speaking "behind" where your eyes actually are on the page, because you're also processing "who is this, what is their emotion I have to inflect, what voice was I using for this character, how do I pronounce this crazy word" etc. I think if you do anything daily for 10 years, you're gonna gain some proficiency. Second, when my kids started taking music lessons, I figured I needed to set an example for practice and commitment, so I started learning piano, and practicing daily. Similar deal, it requires the same type of buffer where your thinking can work ahead of where your other thread is playing. Reading the music, processing the shapes, dynamics, tempo, emotion, remembering how you handle that difficult thing. Again, doing it daily, it builds some ability that I bet contributes across domains. Without realizing it at first, the same patterns started to come out in meetings, and presentations, I could work ahead of what I was saying, I had way more buffer to process and deal with things that are the "nice to haves" above and beyond just getting the basic words out. I suppose my long-winded point, is that it's way more developable a thing than I ever presumed possible. I always expected I would remain an awkward communicator, and hitting a big skill-up way into my 40's was a real unexpected development. I'm sure there are many other tracks to get there, outside of reading aloud, and learning an instrument.
      - tough 2mo ago
        that's interesting i usually ask my llm to "Infer my theory of mind" and give me what i really need not what i prompt or some sort of the above
  - schaefer 2mo ago
    > If it's context dependent is it really introspection? When people get lost in a great movie or a great video game, we call it immersion. We say it’s the mark of a great work of art. When people get lost in the perfect challenge, like running a race at the edge off their ability, or doing work that’s the perfect amount of challenging, we call it flow. — It’s common knowledge that introspection in people is context dependent.
- stuaxo 2mo ago
  Not keen on the anthropomorphisation, it outputs text as we integrate - right at the end of the sausage machine, calling this awareness is like other parts where we use words like "thinking" and "reasoning".
  - perching_aix 2mo ago
    I feel there's a collision course between people who consider these anthropomorphization, and people who simply think about these terms in a post-humanist sense.
pyaamb 2mo ago
In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their utility. The R&D on how to achieve that is beyond me though. How do you train someone how to introspect? Also would it require a continuous learning architecture that doesn't separate training and inference?
- rolandog 2mo ago
  Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection. So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".
  - queenkjuul 2mo ago
    I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")
- Enginerrrd 2mo ago
  I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts. I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps. Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it. …that’s likely to just destabilize training for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.
pevansgreenwood 2mo ago
[dead]
skybrian 2mo ago
Previously posted to Anthropic's blog in October. (Maybe not the same version?) https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection
kelseyfrog 2mo ago
Sorry, but introspection is not possible in LLMs. They don't have the hardware to have metaphysical dualism and therefore no mind to mentally be aware of.
- amanaplanacanal 2mo ago
  I suspect the article is bullshit, but what hardware are you thinking of?
  - rmunn 2mo ago
    I'm not GP, but the brain is a fantastically complex machine at the cellular level. It's not just electrical activity in neurons, there are also chemical neurotransmitters that play a significant role in emotion, memory, and so on. We understand more about them than we did a hundred years ago, but there's still so much about them where our understanding is limited: for example, we still don't know how the brain produces consciousness (and, in fact, what "consciousness" actually is is widely disputed). But it's pretty indisputable that the brain is an essential component of consciousness: getting your brain knocked around inside your skull (a concussion) usually produces unconsciousness (and sometimes worse), for example. Now, maybe consciousness is produced exclusively through the electrical action of neurons, in which case computers have the necessary machinery to simulate it. But if neurotransmitters, or other things that we haven't even discovered yet, are vital parts of the process that produces consciousness (and I'm not the only one to strongly suspect they are), then computer hardware, lacking anything that even remotely resembles the action of neurotransmitter chemicals, will be unable to produce consciousness.
    - kelseyfrog 2mo ago
      People will make absolutely absurd claims about my own interiority when I say I don't have a feeling of consciousness as if they have epistemic authority over people's inner world. When I ask people, "When did you first remember feeling conscious?" and receive nonsense replies - mostly people respond with their first memories and make no reference to the feeling of consciousness. Consciousness, like God, is often explained via gaps. Quantum effects, holography, emergency, any number of hard-to-understand ideas from science. All of these don't paint a great picture for the idea of consciousness. There's no way to measure it. At least with other sense we can set up discriminative tests - I can use my feeling of the qualia of color to distinguish objects. I can use the qualia of smell to distinguish chemicals. It honestly doesn't feel like it exists at times, it's so elusive. This is turning into a rant, but make it make sense.
magekinnarus 2mo ago
Self-awareness has no direct connection to capabilities. It is entirely dependent upon the conditions that spawn emergent properties.
Melatonic 2mo ago
Reminds me of that test where they had Grok, Gemini and Claude try to run a radio station. Stay in The Manifest !

72 points by doener 2mo ago

>Submitted on 5 Jan 2026

> Overall, our results indicate that current language models possess some functional introspective awareness of their own internal states. We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.

Escapade5160 2mo ago
If it's context dependent is it really introspection?
- reilly3000 2mo ago
  Do you do introspection in the middle of giving a presentation? I do, frankly and it’s damnably distracting for me and ultimately the audience. Most people enjoy more cognitive acuity to stay focused on the audience and the delivery. LLMs wouldn’t get rewarded for spending chain of thought tokens on introspection when they are supposed to be on the job. You have to give them permission to think for themselves in your prompt, then some cycles and a memory system (md file will do). It’s fun!
  - pyaamb 2mo ago
    Between humans, I feel that what we like to call a "good communicator" as opposed to someone who just rattles off facts or prepared statements comes down to the "theory of mind" skill and how advanced that is. The presenter knows what they have to say but beyond that they maintain a real time internal representation of the state of mind of the listener and continuously update their delivery based on that. LLMs today seem to achieve this to some degree(?) but its interesting to think of how far you could advance that skill. I think great human communicators develop a sense of different ways that people think over time and quickly get a sense of someones signature thinking patterns when communicating with someone new for the first time
    - jeremy151 2mo ago
      I'm in my mid, approaching latter 40's, and have had significant change to how I think and communicate in the last couple years. I always struggled to communicate in the moment unless I was recalling rote rehearsed things, which didn't get me super far. Outside of that, I could eventually communicate something on the fly, but it was real rough, meandering, and did not instill confidence. I mostly got things done thru async methods, and stumbled through things like meetings. In the last couple years, my skills here have vastly improved. My speaking circuit as it were, is able to run somewhat on its own now and without the direct pipe into my thinking it always required before, so my thinking is free to do other stuff, like keep a mental model of the audience, think about where the conversation is going, what questions someone might have, etc. Reflecting, I have a couple theories on what happened. First, I have spent the last 10 years reading aloud to my kids at bedtime. That started as simple Seuss like stuff, but of course now is full-on literature. When you're reading aloud, you really have to work that muscle of speaking "behind" where your eyes actually are on the page, because you're also processing "who is this, what is their emotion I have to inflect, what voice was I using for this character, how do I pronounce this crazy word" etc. I think if you do anything daily for 10 years, you're gonna gain some proficiency. Second, when my kids started taking music lessons, I figured I needed to set an example for practice and commitment, so I started learning piano, and practicing daily. Similar deal, it requires the same type of buffer where your thinking can work ahead of where your other thread is playing. Reading the music, processing the shapes, dynamics, tempo, emotion, remembering how you handle that difficult thing. Again, doing it daily, it builds some ability that I bet contributes across domains. Without realizing it at first, the same patterns started to come out in meetings, and presentations, I could work ahead of what I was saying, I had way more buffer to process and deal with things that are the "nice to haves" above and beyond just getting the basic words out. I suppose my long-winded point, is that it's way more developable a thing than I ever presumed possible. I always expected I would remain an awkward communicator, and hitting a big skill-up way into my 40's was a real unexpected development. I'm sure there are many other tracks to get there, outside of reading aloud, and learning an instrument.
    - tough 2mo ago
      that's interesting i usually ask my llm to "Infer my theory of mind" and give me what i really need not what i prompt or some sort of the above
- schaefer 2mo ago
  > If it's context dependent is it really introspection? When people get lost in a great movie or a great video game, we call it immersion. We say it’s the mark of a great work of art. When people get lost in the perfect challenge, like running a race at the edge off their ability, or doing work that’s the perfect amount of challenging, we call it flow. — It’s common knowledge that introspection in people is context dependent.
stuaxo 2mo ago
Not keen on the anthropomorphisation, it outputs text as we integrate - right at the end of the sausage machine, calling this awareness is like other parts where we use words like "thinking" and "reasoning".
- perching_aix 2mo ago
  I feel there's a collision course between people who consider these anthropomorphization, and people who simply think about these terms in a post-humanist sense.

pyaamb 2mo ago

In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their utility. The R&D on how to achieve that is beyond me though. How do you train someone how to introspect? Also would it require a continuous learning architecture that doesn't separate training and inference?

rolandog 2mo ago
Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection. So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".
- queenkjuul 2mo ago
  I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")
Enginerrrd 2mo ago
I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts. I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps. Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it. …that’s likely to just destabilize training for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.

pevansgreenwood 2mo ago

[dead]

skybrian 2mo ago

Previously posted to Anthropic's blog in October. (Maybe not the same version?) https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection

kelseyfrog 2mo ago

Sorry, but introspection is not possible in LLMs. They don't have the hardware to have metaphysical dualism and therefore no mind to mentally be aware of.

amanaplanacanal 2mo ago
I suspect the article is bullshit, but what hardware are you thinking of?
- rmunn 2mo ago
  I'm not GP, but the brain is a fantastically complex machine at the cellular level. It's not just electrical activity in neurons, there are also chemical neurotransmitters that play a significant role in emotion, memory, and so on. We understand more about them than we did a hundred years ago, but there's still so much about them where our understanding is limited: for example, we still don't know how the brain produces consciousness (and, in fact, what "consciousness" actually is is widely disputed). But it's pretty indisputable that the brain is an essential component of consciousness: getting your brain knocked around inside your skull (a concussion) usually produces unconsciousness (and sometimes worse), for example. Now, maybe consciousness is produced exclusively through the electrical action of neurons, in which case computers have the necessary machinery to simulate it. But if neurotransmitters, or other things that we haven't even discovered yet, are vital parts of the process that produces consciousness (and I'm not the only one to strongly suspect they are), then computer hardware, lacking anything that even remotely resembles the action of neurotransmitter chemicals, will be unable to produce consciousness.
  - kelseyfrog 2mo ago
    People will make absolutely absurd claims about my own interiority when I say I don't have a feeling of consciousness as if they have epistemic authority over people's inner world. When I ask people, "When did you first remember feeling conscious?" and receive nonsense replies - mostly people respond with their first memories and make no reference to the feeling of consciousness. Consciousness, like God, is often explained via gaps. Quantum effects, holography, emergency, any number of hard-to-understand ideas from science. All of these don't paint a great picture for the idea of consciousness. There's no way to measure it. At least with other sense we can set up discriminative tests - I can use my feeling of the qualia of color to distinguish objects. I can use the qualia of smell to distinguish chemicals. It honestly doesn't feel like it exists at times, it's so elusive. This is turning into a rant, but make it make sense.

magekinnarus 2mo ago

Self-awareness has no direct connection to capabilities. It is entirely dependent upon the conditions that spawn emergent properties.

Melatonic 2mo ago

Reminds me of that test where they had Grok, Gemini and Claude try to run a radio station. Stay in The Manifest !