5 ms·

> AI has no self-awareness What is your mechanistic model of self awareness that yields this conclusion? > It's a tool Does your model suggest that tools can

> AI has no self-awareness What is your mechanistic model of self awareness that yields this conclusion? > It's a tool Does your model suggest that tools can't have self awareness?

deleted 2mo ago
[deleted]
perching_aix 2mo ago
Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*). (*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line"). You'll sometimes catch models mixing up who's who and how many who-s there even are for example. (**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn. I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't? (*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".
- ben_w 2mo ago
  Before reading, know that I am uncertain in either direction. > a hidden representation of self that is continually tended to This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it. > I further fail to identify how it could be hidden or maintained, considering I control like half of it. Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output. > The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line"). While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one". > You'll sometimes catch models mixing up who's who and how many who-s there even are for example. I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that. > If you know of anything like this, your turn now, would be happy to learn. Do you mean like these, or something else? • https://researchportal.hkust.edu.hk/en/publications/decoding-and-controlling-emotion-in-llms-through-human-aligned-re/ https://researchportal.hkust.edu.hk/en/publications/decoding... • https://aclanthology.org/2026.eacl-long.165/ https://aclanthology.org/2026.eacl-long.165/ • https://transformer-circuits.pub/2026/emotions/index.html https://transformer-circuits.pub/2026/emotions/index.html
  - perching_aix 2mo ago
    > This sounds like a personality? Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions. > Perhaps they were losing their self-awareness at the time? I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way. > Do you mean like these, or something else? They're adjacent, but I more meant something like these: https://arxiv.org/abs/2410.03768 https://arxiv.org/abs/2410.03768 https://arxiv.org/abs/2310.18512 https://arxiv.org/abs/2310.18512 https://arxiv.org/abs/2605.26537 https://arxiv.org/abs/2605.26537 So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state. And the best inkling I have towards this is basically: https://www.youtube.com/shorts/WP5_XJY_P0Q https://www.youtube.com/shorts/WP5_XJY_P0Q But then I don't think there's enough covert channel bandwidth in the agent replies for anything interesting like this.
Delk 2mo ago
I honestly don't think a language model is enough for self-awareness, regardless of the exact model of awareness. A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness. Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution. You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival. Even the more abstract of our subjective experiences are in some way rooted in our physical evolution. I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself. AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.
- woeirua 2mo ago
  So… your model is 100% vibes based. Got it.
  - Delk 2mo ago
    I wasn't trying to give a model. The point was that I don't think it's necessary to give one. You didn't address any of what I wrote, let alone provide any counterarguments. Which part of what I wrote do you think was wrong?
    - naasking 2mo ago
      Making definitive claims about whether LLMs do or do not have specific properties absolutely does require precise definitions of those properties that can be used to evaluate those questions. Merely hand waving that LLMs didn't undergo the same evolutionary process is not a definitive argument. For example, the Turing machines and the lambda calculus don't look anything alike, but they are fundamentally interconvertible, and so in a real sense they are fundamentally equivalent. Without a model, all of your arguments are completely unconvincing for exactly the same reasons, eg. that there may exist many paths to fundamentally equivalent ends.

> AI has no self-awareness What is your mechanistic model of self awareness that yields this conclusion? > It's a tool Does your model suggest that tools can't have self awareness?

deleted 2mo ago

[deleted]

perching_aix 2mo ago

Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*). (*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line"). You'll sometimes catch models mixing up who's who and how many who-s there even are for example. (**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn. I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't? (*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".

ben_w 2mo ago
Before reading, know that I am uncertain in either direction. > a hidden representation of self that is continually tended to This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it. > I further fail to identify how it could be hidden or maintained, considering I control like half of it. Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output. > The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line"). While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one". > You'll sometimes catch models mixing up who's who and how many who-s there even are for example. I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that. > If you know of anything like this, your turn now, would be happy to learn. Do you mean like these, or something else? • https://researchportal.hkust.edu.hk/en/publications/decoding-and-controlling-emotion-in-llms-through-human-aligned-re/ https://researchportal.hkust.edu.hk/en/publications/decoding... • https://aclanthology.org/2026.eacl-long.165/ https://aclanthology.org/2026.eacl-long.165/ • https://transformer-circuits.pub/2026/emotions/index.html https://transformer-circuits.pub/2026/emotions/index.html
- perching_aix 2mo ago
  > This sounds like a personality? Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions. > Perhaps they were losing their self-awareness at the time? I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way. > Do you mean like these, or something else? They're adjacent, but I more meant something like these: https://arxiv.org/abs/2410.03768 https://arxiv.org/abs/2410.03768 https://arxiv.org/abs/2310.18512 https://arxiv.org/abs/2310.18512 https://arxiv.org/abs/2605.26537 https://arxiv.org/abs/2605.26537 So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state. And the best inkling I have towards this is basically: https://www.youtube.com/shorts/WP5_XJY_P0Q https://www.youtube.com/shorts/WP5_XJY_P0Q But then I don't think there's enough covert channel bandwidth in the agent replies for anything interesting like this.

Delk 2mo ago

I honestly don't think a language model is enough for self-awareness, regardless of the exact model of awareness. A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness. Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution. You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival. Even the more abstract of our subjective experiences are in some way rooted in our physical evolution. I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself. AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.

woeirua 2mo ago
So… your model is 100% vibes based. Got it.
- Delk 2mo ago
  I wasn't trying to give a model. The point was that I don't think it's necessary to give one. You didn't address any of what I wrote, let alone provide any counterarguments. Which part of what I wrote do you think was wrong?
  - naasking 2mo ago
    Making definitive claims about whether LLMs do or do not have specific properties absolutely does require precise definitions of those properties that can be used to evaluate those questions. Merely hand waving that LLMs didn't undergo the same evolutionary process is not a definitive argument. For example, the Turing machines and the lambda calculus don't look anything alike, but they are fundamentally interconvertible, and so in a real sense they are fundamentally equivalent. Without a model, all of your arguments are completely unconvincing for exactly the same reasons, eg. that there may exist many paths to fundamentally equivalent ends.