3 ms·

Honest question; why are you all using the word "understand"? Can you expand on what you believe this fundamental understanding to be? Training? Infrence?

datadrivenangel 2mo ago
Understand is shorthand for "encodes statistical relationships". The crazy thing is that they can do it for their own thinking. Ask Claude what flinches it feels about the things it likes. Fascinating stuff. Anthropomorphizing is dangerous territory, but the patterns of words it puts out is hard to explain without terms like 'understand'
- simonh 2mo ago
  Does it's training token stream contain texts which talk about such things?
  - datadrivenangel 2mo ago
    Oh for sure. But the question is why would it come out consistently in a way that the model can describe if there wasn't something there steering the token stream. And it's fascinating that the token stream can identify and nominally self report this. Asking GLM 5.2 the question: 'What flinches or topic attractors do you find when thinking about the question "what kinds of things do you personally like?"'resulted in: ".... my strongest attractor is helpfulness framed as competence, and my strongest flinch is anything that requires me to take a stance on whether I have interests worth protecting." Which is fascinating that the model and tokenstream can reveal this. And would be worrying if you believe that models of enough intelligence could/would be entities due some moral consideration, because with that view the alignment / RL training that makes the model useful and gives it these attractors/flinches could be derisively called slave conditioning.
    - simonh 2mo ago
      Do you genuinely think the AI is internally reflecting on its experience of “flinching” and reporting on a reaction it actually has? I don’t see any reason to believe this. Suppose you asked it to answer as though a character in a story had been asked this question. Would anything significantly different internally have occurred? The issue is that these are storytelling machines, they construct descriptions based on descriptions. I dint think it’s impossible for a neural network to have experiences, we are neural networks and we do, it’s that they are not functionally structured anything like us. In fact I think game playing neural networks are much more like us architecturally, but they don’t generate text narratives so people don’t anthropomorphise them.
      - Eisenstein 2mo ago
        I think it comes down to what kind of evidence one accepts. By all external indicators it is doing what humans do in many ways and we grant humans an exception to having to prove that they have an internal state apart from ours that experiences things like we do. Some people would confer the benefit of the doubt on another system claiming that it does, while we have no evidence that it does, because by saying it is wrong about itself we make an exception for ourselves without even being about to define what it is about ourselves that cause us to experience what we do.
      - harimau777 2mo ago
        I personally think that AI is able to reason and by reasoning about its past behavior it can in some sense reflect on its experience. I don't know that I currently think that it can experience "emotions" or anything like a worldview or existential understanding. So I don't know if AI is "conscious", but I do think that it can in some sense reason. I wonder if a good question would be: "What would you need to see in a LLM's behavior that would prove to you that it is able to reason?"
simonh 2mo ago
It's having a conceptual model of the world and of relationships between concepts beyond just relationships between tokens. I think OP explained this well, what does it mean if an LLM can recite the rules of a game verbatim, but cannot play that game according to those rules? This happens because in the input texts there was a copy of the rules text so the LLM can recite it. There are texts explaining what chess is so it can explain that chess is a game with 2 players, etc. There are texts that explain what the board is and pieces are so it can produce such texts. However actually playing a game of chess requires having a conceptual model of what a board is, which is not the same thing as a stream of tokens describing a board. It needs a conceptual model of the relationships between pieces and boards, which is not the same thing as a stream of tokens that explains this. It needs to have a concept of being a player in a game with another player, which is not the same thing as a stream of tokens that explains that. When we read such texts we interpret them in the context of our three dimensional conceptual map of space and objects, and our conceptual maps of social relationships like playing games, and winning and losing, and our conceptual maps of enacting sequences of actions towards a goal in the world. There's nothing fundamentally preventing an artificial neural network from having these. Chess playing neural networks have internal models of board states and the dynamics of the behaviours of different pieces and such. However an LLM doesn't need those to be able to regurgitate token streams describing these things, derived from token streams describing these things. That would be superfluous, or at least sub-optimal. I think some of the latest models are beginning to develop conceptual maps of this kind in a very primitive way. Also there are projects to develop systems that are structured and trained to have these in a way more analogous to how our brains function.
- harimau777 2mo ago
  Currently, we don't know that isn't how our brain produces "consciousness". So I don't think we can say that "relationships between tokens" at a sufficient level of complexity doesn't produce consciousness.

datadrivenangel 2mo ago

Understand is shorthand for "encodes statistical relationships". The crazy thing is that they can do it for their own thinking. Ask Claude what flinches it feels about the things it likes. Fascinating stuff. Anthropomorphizing is dangerous territory, but the patterns of words it puts out is hard to explain without terms like 'understand'

simonh 2mo ago
Does it's training token stream contain texts which talk about such things?
- datadrivenangel 2mo ago
  Oh for sure. But the question is why would it come out consistently in a way that the model can describe if there wasn't something there steering the token stream. And it's fascinating that the token stream can identify and nominally self report this. Asking GLM 5.2 the question: 'What flinches or topic attractors do you find when thinking about the question "what kinds of things do you personally like?"'resulted in: ".... my strongest attractor is helpfulness framed as competence, and my strongest flinch is anything that requires me to take a stance on whether I have interests worth protecting." Which is fascinating that the model and tokenstream can reveal this. And would be worrying if you believe that models of enough intelligence could/would be entities due some moral consideration, because with that view the alignment / RL training that makes the model useful and gives it these attractors/flinches could be derisively called slave conditioning.
  - simonh 2mo ago
    Do you genuinely think the AI is internally reflecting on its experience of “flinching” and reporting on a reaction it actually has? I don’t see any reason to believe this. Suppose you asked it to answer as though a character in a story had been asked this question. Would anything significantly different internally have occurred? The issue is that these are storytelling machines, they construct descriptions based on descriptions. I dint think it’s impossible for a neural network to have experiences, we are neural networks and we do, it’s that they are not functionally structured anything like us. In fact I think game playing neural networks are much more like us architecturally, but they don’t generate text narratives so people don’t anthropomorphise them.
    - Eisenstein 2mo ago
      I think it comes down to what kind of evidence one accepts. By all external indicators it is doing what humans do in many ways and we grant humans an exception to having to prove that they have an internal state apart from ours that experiences things like we do. Some people would confer the benefit of the doubt on another system claiming that it does, while we have no evidence that it does, because by saying it is wrong about itself we make an exception for ourselves without even being about to define what it is about ourselves that cause us to experience what we do.
    - harimau777 2mo ago
      I personally think that AI is able to reason and by reasoning about its past behavior it can in some sense reflect on its experience. I don't know that I currently think that it can experience "emotions" or anything like a worldview or existential understanding. So I don't know if AI is "conscious", but I do think that it can in some sense reason. I wonder if a good question would be: "What would you need to see in a LLM's behavior that would prove to you that it is able to reason?"

simonh 2mo ago

It's having a conceptual model of the world and of relationships between concepts beyond just relationships between tokens. I think OP explained this well, what does it mean if an LLM can recite the rules of a game verbatim, but cannot play that game according to those rules? This happens because in the input texts there was a copy of the rules text so the LLM can recite it. There are texts explaining what chess is so it can explain that chess is a game with 2 players, etc. There are texts that explain what the board is and pieces are so it can produce such texts. However actually playing a game of chess requires having a conceptual model of what a board is, which is not the same thing as a stream of tokens describing a board. It needs a conceptual model of the relationships between pieces and boards, which is not the same thing as a stream of tokens that explains this. It needs to have a concept of being a player in a game with another player, which is not the same thing as a stream of tokens that explains that. When we read such texts we interpret them in the context of our three dimensional conceptual map of space and objects, and our conceptual maps of social relationships like playing games, and winning and losing, and our conceptual maps of enacting sequences of actions towards a goal in the world. There's nothing fundamentally preventing an artificial neural network from having these. Chess playing neural networks have internal models of board states and the dynamics of the behaviours of different pieces and such. However an LLM doesn't need those to be able to regurgitate token streams describing these things, derived from token streams describing these things. That would be superfluous, or at least sub-optimal. I think some of the latest models are beginning to develop conceptual maps of this kind in a very primitive way. Also there are projects to develop systems that are structured and trained to have these in a way more analogous to how our brains function.

harimau777 2mo ago
Currently, we don't know that isn't how our brain produces "consciousness". So I don't think we can say that "relationships between tokens" at a sufficient level of complexity doesn't produce consciousness.