5 ms·

LLMs can't explain their past behavior because any time you ask them to, all they're really doing is reading the previous context from a previous inference and

LLMs can't explain their past behavior because any time you ask them to, all they're really doing is reading the previous context from a previous inference and looking for any reason they can think of that they might have said that, so to speak. That is, they have absolutely no genuine recollection of what they were thinking at the time they said something in the past. Even with "Tree of Thought" approaches all you're doing is recording past conversations, states, and contexts, and your new inference asking for the "justification" of that, will be a similarly totally fake justification, because as I said they have no memory but only context. In my own app I can switch to a different LLM right in the middle of a conversation and the new LLM will just continue to always think it said everything in the prior context even though that's not the case.

Terretta 2y ago
> the new LLM will just continue to always think Even when you know, as you do, it's tough to avoid such characterizations.
- 01HNNWZ0MV43FF 2y ago
  It might be fair to say they think but they can't directly introspect the thinking process. They can only confabulate reasoning post-hoc, or use science to try to understand themselves from the outside. Same as humans.
  - stackghost 2y ago
    >Same as humans Hard disagree. LLMs are very accurately described by the "stochastic parrot" analogy that gets thrown around a lot. They do not "think" like humans at all, even if we use the word "think" because it's convenient.
    - Der_Einzige 2y ago
      The more tokens spent on getting a result, the more likely the result is to be accurate. “If it walks like a duck, quacks like a duck” etc
      - deleted 2y ago
        [deleted]
      - bredren 2y ago
        I suspect gp means thinking in a way that is different from being able to reason. The ability to use any manner of inference or logic to arrive at correct answers does not constitute thought, even if the question was hard. But in general I’m in agreement about the duck. Existing LLMs can already “think” in a non-philosophical sense far better and faster than many adults walking around. Most folks do not Consider the Lobster anyway and would likely not pay much attention to the details of its difference from a brain.
    - quantadev 2y ago
      I think the phrase "stochastic parrot" is misleading and has fooled millions of people into thinking that LLMs can't do genuine reasoning about situations they've never seen before, nor been trained on, which is wrong because LLMs definitely are doing genuine intelligent reasoning. What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during inference. I mean like a number of facts larger than the number of molecules in the known universe. A googolplex more facts than the sum of all of humanity has ever "thought".
      - bobsomers 2y ago
        > I think the phrase "stochastic parrot" is misleading and has fooled millions of people into thinking that LLMs can't do genuine reasoning about situations they've never seen before, nor been trained on, which is wrong because LLMs definitely are doing genuine intelligent reasoning. Can you describe specifically which part of an LLM architecture does the "reasoning" and how it works? Because every architecture I'm familiar with is literally just a fancy way of predicting the likelihood of the next token given the stream of previous tokens and the known distribution of token based on the training data. This is not reasoning. This is simple statistical prediction, making the "stochastic parrot" analogy actually quite accurate. > What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during inference. I mean like a number of facts larger than the number of molecules in the known universe. A googolplex more facts than the sum of all of humanity has ever "thought". Sort of. It's like an extremely lossy compression process which has absolutely no guarantee that "truth" was maintained in the process. Also I'm extremely dubious of your claim that it can accurately encode "a number of facts larger than the molecules in the known universe" given that it's trivially easy to get the best LLMs to give you an incorrect answer to a question any human would easily get right.
        quantadev 2y ago
        The "reasoning" is an emergent property that no one understands yet. Yes we do all the training only to try to predict the next word (i.e. train to do word prediction), yet with enough training data, then at some scale (GPT 3.5ish) the embedding vectors in semantic space begin to build a geometric scaffolding in into the weights, for lack of a better way to phrase it. If you know about facts like (Vector(man) minus Vector(woman) equals Vector(king) minus Vector(queen)), that's an indication that this "scaffolding" is taking shape. It means the "concept of gender" has a "direction" in the roughly 4,000 dimensional vector "space". This vector behaves geometrically, so that vectors behave in vector space as if it was a geometric space of sorts (there are directions and distances), even though there's no true space coordinates, just logical "directions". Mankind doesn't quite yet understand the "Geometry of Logic". LLMs prove we don't. I think it's a new math field to be invented. As far as the actual number of "facts" contained in an LLM, I think you have to consider something that's a function of the number of bits in an entire model, and ask how many "states" can that store, as a rough approximation from an entropy standpoint. But these aren't pure facts. They're reasoning. I guess you can call reasoning something like "fuzzy facts", so there's a bit of uncertainty to each one of them. LLMs don't store facts, they store fuzzy reasoning. But I call it "factual" when an LLM fixes a bug in my code, or correctly states some piece of knowledge.
    - MobiusHorizons 2y ago
      I tend to agree that LLMs are not thinking in the way that we usually mean it when referring to human thinking. However I think it is dangerous to assert on the capabilities of a system based on the structure seemingly imposed by its API. Take for example the instruction set of a CPU. One could argue that a cpu only has N registers or can process only one instruction at a time because the instruction set only contains N registers, and processes instruction linearly. But on any modern application processor, the register renamer allows many more physical registers to be allocated than can be names in the ISA, and instructions are dispatched in parallel to mitigate memory latency and increase throughput. What I mean is that an LLM is not a stochastic parrot because of its API, but rather because it does not outdo a stochastic parrot when tested. This could change though. LLMs could think in full sentences and spoon feed them to us one token at a time, rewording as necessary to provide a few possibilities. They could memorize the letters in each token and count letters correctly despite the limitations imposed by the API.
- quantadev 2y ago
  I didn't use the word "think" in the human sense of the word, or even in the consciousness/qualia sense of the word. I of course only meant inference (Matrix multiplications) What I like to say however is that LLMs are doing genuine "reasoning"; and what the emergent "intelligent" behavior of LLMs has proven to mankind is that "reasoning" and "consciousness/qualia" are separable things (not identical), and I think as recently as 2021 we didn't really know/expect that to be the case.
  - adastra22 2y ago
    What other sense of the word would there be? That’s essentially what’s going on in your brain too.
    - quantadev 2y ago
      I always assume some people might define "thinking" as requiring "consciousness", because up until about 2022 pretty much every human on the planet did.
      - adastra22 2y ago
        So? Are you saying consciousness isn’t a solely mechanistic artifact of neural processing in your brain? What else would it be?
        quantadev 2y ago
        Consciousness/qualia actually correlates stronger to brain waves than to synaptic activity, and for that and a lot of other reasons (that I'm presently writing a paper on) I think consciousness is a phenomena made up of waves, and while it may indeed be "substrate independent" it's not just purely calculations, and therefore cannot be done in computers. Even a perfect simulation of a human brain in a computer would be nothing but a "predictor" of behavior, rather than a "feeler" of behavior.
        2y ago
resonious 2y ago
Split brain experiments show there's a good amount of evidence that humans also largely operate in this way.
- quantadev 2y ago
  You're right. Humans have short term memory, so we can genuinely recall our reasoning, but at the same time, once an action (or even a thought) has been had, we begin weaving a new narrative about why we did that thing or had that thought, that often has no resemblance to actual truth.
- greyface- 2y ago
  Split brain experiments show that humans missing a functional corpus callosum largely operate in this way. I think it's an open question whether all humans operate in this way, or if communication between the hemispheres is necessary for 'real' explanations of behavior. The data we have at this point would support either conclusion.
  - famouswaffles 2y ago
    Split-brain aside, there are other studies suggesting a huge amount of post-hoc rationalization for human choices and decision making. Like it seems obvious and intuitive that preferences form choices but it turns out that choices shape preferences just as easily. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://journals.sagepub.com/doi/abs/10.1177/0956797620954491 https://journals.sagepub.com/doi/abs/10.1177/095679762095449... At this point, I think there is little doubt humans regularly fabricate explanations of their behaviour unknowingly. It's just a question of how much and to what degrees.
- samtho 2y ago
  This is very reductive and gross misrepresentation of split brain experiments. Humans have various level of memory (working, short term, long term), the ability to experience meta cognition, and the autonomy to make a decision - all of which (and much, much more) informs our behavior and are also things that an LLM fundamentally does not have. These “studies” are painfully naive at best and are at worse an intentional misinformation campaign used to bolster the value of LLM past what they are useful for.
owenpalmer 2y ago
> That is, they have absolutely no genuine recollection of what they were thinking at the time they said something in the past. I really like this point. Imagine if model observability got to the point where a model could self-observe after running a response. Not only would it read the context, it could observe it's own thoughts from the past.
mattmaroon 2y ago
To be honest, there is a lot of reason to believe that this is more or less how humans work also. We might be better at figuring out why we did something, and we might be more deliberate in some cases. But for the most part I think and then logically justify their actions.
igornadj 2y ago
Isn't this covered in the first FAQ in the OP?

Terretta 2y ago

> the new LLM will just continue to always think Even when you know, as you do, it's tough to avoid such characterizations.

01HNNWZ0MV43FF 2y ago
It might be fair to say they think but they can't directly introspect the thinking process. They can only confabulate reasoning post-hoc, or use science to try to understand themselves from the outside. Same as humans.
- stackghost 2y ago
  >Same as humans Hard disagree. LLMs are very accurately described by the "stochastic parrot" analogy that gets thrown around a lot. They do not "think" like humans at all, even if we use the word "think" because it's convenient.
  - Der_Einzige 2y ago
    The more tokens spent on getting a result, the more likely the result is to be accurate. “If it walks like a duck, quacks like a duck” etc
    - deleted 2y ago
      [deleted]
    - bredren 2y ago
      I suspect gp means thinking in a way that is different from being able to reason. The ability to use any manner of inference or logic to arrive at correct answers does not constitute thought, even if the question was hard. But in general I’m in agreement about the duck. Existing LLMs can already “think” in a non-philosophical sense far better and faster than many adults walking around. Most folks do not Consider the Lobster anyway and would likely not pay much attention to the details of its difference from a brain.
  - quantadev 2y ago
    I think the phrase "stochastic parrot" is misleading and has fooled millions of people into thinking that LLMs can't do genuine reasoning about situations they've never seen before, nor been trained on, which is wrong because LLMs definitely are doing genuine intelligent reasoning. What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during inference. I mean like a number of facts larger than the number of molecules in the known universe. A googolplex more facts than the sum of all of humanity has ever "thought".
    - bobsomers 2y ago
      > I think the phrase "stochastic parrot" is misleading and has fooled millions of people into thinking that LLMs can't do genuine reasoning about situations they've never seen before, nor been trained on, which is wrong because LLMs definitely are doing genuine intelligent reasoning. Can you describe specifically which part of an LLM architecture does the "reasoning" and how it works? Because every architecture I'm familiar with is literally just a fancy way of predicting the likelihood of the next token given the stream of previous tokens and the known distribution of token based on the training data. This is not reasoning. This is simple statistical prediction, making the "stochastic parrot" analogy actually quite accurate. > What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during inference. I mean like a number of facts larger than the number of molecules in the known universe. A googolplex more facts than the sum of all of humanity has ever "thought". Sort of. It's like an extremely lossy compression process which has absolutely no guarantee that "truth" was maintained in the process. Also I'm extremely dubious of your claim that it can accurately encode "a number of facts larger than the molecules in the known universe" given that it's trivially easy to get the best LLMs to give you an incorrect answer to a question any human would easily get right.
      - quantadev 2y ago
        The "reasoning" is an emergent property that no one understands yet. Yes we do all the training only to try to predict the next word (i.e. train to do word prediction), yet with enough training data, then at some scale (GPT 3.5ish) the embedding vectors in semantic space begin to build a geometric scaffolding in into the weights, for lack of a better way to phrase it. If you know about facts like (Vector(man) minus Vector(woman) equals Vector(king) minus Vector(queen)), that's an indication that this "scaffolding" is taking shape. It means the "concept of gender" has a "direction" in the roughly 4,000 dimensional vector "space". This vector behaves geometrically, so that vectors behave in vector space as if it was a geometric space of sorts (there are directions and distances), even though there's no true space coordinates, just logical "directions". Mankind doesn't quite yet understand the "Geometry of Logic". LLMs prove we don't. I think it's a new math field to be invented. As far as the actual number of "facts" contained in an LLM, I think you have to consider something that's a function of the number of bits in an entire model, and ask how many "states" can that store, as a rough approximation from an entropy standpoint. But these aren't pure facts. They're reasoning. I guess you can call reasoning something like "fuzzy facts", so there's a bit of uncertainty to each one of them. LLMs don't store facts, they store fuzzy reasoning. But I call it "factual" when an LLM fixes a bug in my code, or correctly states some piece of knowledge.
  - MobiusHorizons 2y ago
    I tend to agree that LLMs are not thinking in the way that we usually mean it when referring to human thinking. However I think it is dangerous to assert on the capabilities of a system based on the structure seemingly imposed by its API. Take for example the instruction set of a CPU. One could argue that a cpu only has N registers or can process only one instruction at a time because the instruction set only contains N registers, and processes instruction linearly. But on any modern application processor, the register renamer allows many more physical registers to be allocated than can be names in the ISA, and instructions are dispatched in parallel to mitigate memory latency and increase throughput. What I mean is that an LLM is not a stochastic parrot because of its API, but rather because it does not outdo a stochastic parrot when tested. This could change though. LLMs could think in full sentences and spoon feed them to us one token at a time, rewording as necessary to provide a few possibilities. They could memorize the letters in each token and count letters correctly despite the limitations imposed by the API.
quantadev 2y ago
I didn't use the word "think" in the human sense of the word, or even in the consciousness/qualia sense of the word. I of course only meant inference (Matrix multiplications) What I like to say however is that LLMs are doing genuine "reasoning"; and what the emergent "intelligent" behavior of LLMs has proven to mankind is that "reasoning" and "consciousness/qualia" are separable things (not identical), and I think as recently as 2021 we didn't really know/expect that to be the case.
- adastra22 2y ago
  What other sense of the word would there be? That’s essentially what’s going on in your brain too.
  - quantadev 2y ago
    I always assume some people might define "thinking" as requiring "consciousness", because up until about 2022 pretty much every human on the planet did.
    - adastra22 2y ago
      So? Are you saying consciousness isn’t a solely mechanistic artifact of neural processing in your brain? What else would it be?
      - quantadev 2y ago
        Consciousness/qualia actually correlates stronger to brain waves than to synaptic activity, and for that and a lot of other reasons (that I'm presently writing a paper on) I think consciousness is a phenomena made up of waves, and while it may indeed be "substrate independent" it's not just purely calculations, and therefore cannot be done in computers. Even a perfect simulation of a human brain in a computer would be nothing but a "predictor" of behavior, rather than a "feeler" of behavior.
        2y ago

resonious 2y ago

Split brain experiments show there's a good amount of evidence that humans also largely operate in this way.

quantadev 2y ago
You're right. Humans have short term memory, so we can genuinely recall our reasoning, but at the same time, once an action (or even a thought) has been had, we begin weaving a new narrative about why we did that thing or had that thought, that often has no resemblance to actual truth.
greyface- 2y ago
Split brain experiments show that humans missing a functional corpus callosum largely operate in this way. I think it's an open question whether all humans operate in this way, or if communication between the hemispheres is necessary for 'real' explanations of behavior. The data we have at this point would support either conclusion.
- famouswaffles 2y ago
  Split-brain aside, there are other studies suggesting a huge amount of post-hoc rationalization for human choices and decision making. Like it seems obvious and intuitive that preferences form choices but it turns out that choices shape preferences just as easily. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://journals.sagepub.com/doi/abs/10.1177/0956797620954491 https://journals.sagepub.com/doi/abs/10.1177/095679762095449... At this point, I think there is little doubt humans regularly fabricate explanations of their behaviour unknowingly. It's just a question of how much and to what degrees.
samtho 2y ago
This is very reductive and gross misrepresentation of split brain experiments. Humans have various level of memory (working, short term, long term), the ability to experience meta cognition, and the autonomy to make a decision - all of which (and much, much more) informs our behavior and are also things that an LLM fundamentally does not have. These “studies” are painfully naive at best and are at worse an intentional misinformation campaign used to bolster the value of LLM past what they are useful for.

owenpalmer 2y ago

> That is, they have absolutely no genuine recollection of what they were thinking at the time they said something in the past. I really like this point. Imagine if model observability got to the point where a model could self-observe after running a response. Not only would it read the context, it could observe it's own thoughts from the past.

mattmaroon 2y ago

To be honest, there is a lot of reason to believe that this is more or less how humans work also. We might be better at figuring out why we did something, and we might be more deliberate in some cases. But for the most part I think and then logically justify their actions.

igornadj 2y ago

Isn't this covered in the first FAQ in the OP?