3 ms·

>Chatbot LLM's don't have metacognition. The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially

>Chatbot LLM's don't have metacognition. The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".

gizajob 2y ago
superficially being the operative word.
kaoD 2y ago
> at least a simulacrum that is superficially indistinguishable from metacognition That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer. It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
- thesumofall 2y ago
  I feel it’s very rare that people openly and consciously reflect about what they do or do not know. And even then I find it questionable if we are really capable of identifying the difference reliably. However, most of the time I feel we just make it up - esp in none-trivial areas. Just think of the average business meeting
- gus_massa 2y ago
  > an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". I've seen that in humans too. For example after gradding an exam, a student may come and explain why they made a mistake and why they intended to do and why we shoud increase the grade. Most of the times the new explanation is as bad as the original one.
  - kaoD 2y ago
    Yes, humans can bullshit too. Is that the standard to which we want to hold our AI? The question is not if humans do bullshit like LLMs, it's whether AI can think like humans.
    - rowanG077 2y ago
      I don't think the question is AI can think like humans. The question is whether AI can perform task like humans. We don't even know how humans think. Even asking the question whether AI can think like humans at this point is pretty non-sensical.
- Terr_ 2y ago
  Right, to express it another way: The real-world LLM just makes documents longer. It is being run against a document which resembles a movie script, where "User says X, Computer says" is inserted whenever I type X, and then the LLM just makes it a bit longer to complete the line for the "Computer" character. These models marketed as "reasoning" are just changing the style of the script to film noir detective, where the protagonist "Computer" has extra observations and commentary that aren't "spoken" to other characters. While that change may help keep the story on track, it does not affect the fundamental nature of the "thinking" going on: It's still just growing a document and acting out whatever story emerges.
  - hnaccount_rng 2y ago
    That’s a very good analogy, which works astonishingly well in giving people a reasonable expectation. The question I’m having though is: How different are we from that? The nature of a Markov-Chain is that you can describe just about anything with it. The following description is correct, isn’t it? Our “thoughts” are generated as some function of our integrated past sensory input (scare quotes because I don’t want to talk about what exactly a thought is)
    - Terr_ 2y ago
      > The question I’m having though is: How different are we from that? It's obvious we are different, but if we could answer exactly how with sufficient rigor, we would already have better AI and be asking different questions. :p I readily admit that LLMs are an exciting potential piece of a much bigger puzzle, but this could easily be like trying to parse HTML only using regular expressions: It works great on trivial input, but no amount of minor would let it truly solve the problem, because it lacks some higher structure of organization or meaning.
      - hnaccount_rng 2y ago
        I too thought that it was obvious. But then I spend a lot of time in neuroscience discussions… and… it got less obvious
mercer 2y ago
I thought the innovation was that these phrases are 'manually' inserted to force the model to consider different options. Did I get that wrong?
- tmnvdb 2y ago
  For R1-Zero they did RL on two properties: a) there is a 'thinking' box and an 'answer box' (format constraint) b) the answer box has a correct answer Note that there is nothing about the contents of the thinking box in the reinforcement learning. Only that there is such a box. They then observe that during RL, the model will start generating more and more stuff in the thinking box. In essence this is the emergence of using more test-time compute to improve answers. When reading out the thinking box, they found the model reflecting on its own answers, going back and changing its mind, and reflecting on the question, etc, similar to what we can see with R1 now. These are fully emergent phenomenon, without any prompting to do such a thing! (The reasoning output would sometimes be a bit garbled and switch languages randomly, so for R1 they added some constraints to make the thinking content intelligble to humans. This actually made the model slightly worse at answering questions correctly.)
  - mercer 2y ago
    Wow. That's even cooler (and somehow simpler) than I thought it was.
ianmcnaney 2y ago
Do they? Try asking copilot whether thequantuminsider.com is a credible source, and to cite references. It currently answers with three references to thequantuminsider.com. That's clearly not a valid answer, and if you point out that it's invalid to use a source to support its own credibility it will agree. It used to come back with three more references to thequantuminsider.com but when I tried just now it said something about looking for more information and appeared to hang. After asking how long this would take it said something like "not long". After a few more minutes I said "there aren't any credible independent sources, are there?" At that point it agreed that there are not, but that the site is popular in the quantum computing community which lends it credibility. It then provided three links to thequantuminsider.com in support of this statement. It just goes round and round. Metacognition indeed. edit: but why do plants crave Brawndo?

gizajob 2y ago

superficially being the operative word.

kaoD 2y ago

> at least a simulacrum that is superficially indistinguishable from metacognition That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer. It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.

thesumofall 2y ago
I feel it’s very rare that people openly and consciously reflect about what they do or do not know. And even then I find it questionable if we are really capable of identifying the difference reliably. However, most of the time I feel we just make it up - esp in none-trivial areas. Just think of the average business meeting
gus_massa 2y ago
> an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". I've seen that in humans too. For example after gradding an exam, a student may come and explain why they made a mistake and why they intended to do and why we shoud increase the grade. Most of the times the new explanation is as bad as the original one.
- kaoD 2y ago
  Yes, humans can bullshit too. Is that the standard to which we want to hold our AI? The question is not if humans do bullshit like LLMs, it's whether AI can think like humans.
  - rowanG077 2y ago
    I don't think the question is AI can think like humans. The question is whether AI can perform task like humans. We don't even know how humans think. Even asking the question whether AI can think like humans at this point is pretty non-sensical.
Terr_ 2y ago
Right, to express it another way: The real-world LLM just makes documents longer. It is being run against a document which resembles a movie script, where "User says X, Computer says" is inserted whenever I type X, and then the LLM just makes it a bit longer to complete the line for the "Computer" character. These models marketed as "reasoning" are just changing the style of the script to film noir detective, where the protagonist "Computer" has extra observations and commentary that aren't "spoken" to other characters. While that change may help keep the story on track, it does not affect the fundamental nature of the "thinking" going on: It's still just growing a document and acting out whatever story emerges.
- hnaccount_rng 2y ago
  That’s a very good analogy, which works astonishingly well in giving people a reasonable expectation. The question I’m having though is: How different are we from that? The nature of a Markov-Chain is that you can describe just about anything with it. The following description is correct, isn’t it? Our “thoughts” are generated as some function of our integrated past sensory input (scare quotes because I don’t want to talk about what exactly a thought is)
  - Terr_ 2y ago
    > The question I’m having though is: How different are we from that? It's obvious we are different, but if we could answer exactly how with sufficient rigor, we would already have better AI and be asking different questions. :p I readily admit that LLMs are an exciting potential piece of a much bigger puzzle, but this could easily be like trying to parse HTML only using regular expressions: It works great on trivial input, but no amount of minor would let it truly solve the problem, because it lacks some higher structure of organization or meaning.
    - hnaccount_rng 2y ago
      I too thought that it was obvious. But then I spend a lot of time in neuroscience discussions… and… it got less obvious

mercer 2y ago

I thought the innovation was that these phrases are 'manually' inserted to force the model to consider different options. Did I get that wrong?

tmnvdb 2y ago
For R1-Zero they did RL on two properties: a) there is a 'thinking' box and an 'answer box' (format constraint) b) the answer box has a correct answer Note that there is nothing about the contents of the thinking box in the reinforcement learning. Only that there is such a box. They then observe that during RL, the model will start generating more and more stuff in the thinking box. In essence this is the emergence of using more test-time compute to improve answers. When reading out the thinking box, they found the model reflecting on its own answers, going back and changing its mind, and reflecting on the question, etc, similar to what we can see with R1 now. These are fully emergent phenomenon, without any prompting to do such a thing! (The reasoning output would sometimes be a bit garbled and switch languages randomly, so for R1 they added some constraints to make the thinking content intelligble to humans. This actually made the model slightly worse at answering questions correctly.)
- mercer 2y ago
  Wow. That's even cooler (and somehow simpler) than I thought it was.

ianmcnaney 2y ago

Do they? Try asking copilot whether thequantuminsider.com is a credible source, and to cite references. It currently answers with three references to thequantuminsider.com. That's clearly not a valid answer, and if you point out that it's invalid to use a source to support its own credibility it will agree. It used to come back with three more references to thequantuminsider.com but when I tried just now it said something about looking for more information and appeared to hang. After asking how long this would take it said something like "not long". After a few more minutes I said "there aren't any credible independent sources, are there?" At that point it agreed that there are not, but that the site is popular in the quantum computing community which lends it credibility. It then provided three links to thequantuminsider.com in support of this statement. It just goes round and round. Metacognition indeed. edit: but why do plants crave Brawndo?