3 ms·

> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying

> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension. I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.

Diogenesian 4mo ago
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
- throwaway7356 4mo ago
  > yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability Where can I access such a Lisp expert system? If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
PaulDavisThe1st 4mo ago
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
- naasking 4mo ago
  I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
  - PaulDavisThe1st 4mo ago
    I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
    - naasking 4mo ago
      I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc. I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.
      - PaulDavisThe1st 4mo ago
        I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often. The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way. I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
        naasking 4mo ago
        > The child knows how to catch the ball, without understanding. I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way. > I don't think that it is useful to say that LLMs understand anything they say, or that we say to them. This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.
andy99 4mo ago
I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns. There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
- emtel 4mo ago
  > should I walk or drive to the car wash there is an entire genre of riddles based on this kind of misdirection that works on humans, such as "A plane crashed on the border or US and Canada. Where do they bury the survivors?" As with humans, comprehension is not a binary property of the agent - it is a quality that can be present in some situations and absent in others. LLMs may emit correct outputs sometimes because they do comprehend the input, and emit incorrect outputs in other cases when they do not comprehend the input. In order to show that LLMs can't comprehend, we'd have to show that there are no (or at least very few) situations in which they exhibit comprehension, not show that there are some situations in which they don't.
27183 4mo ago
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. Is it possible you're making the following error described in the article? > The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people. Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation? I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
- SpicyLemonZest 4mo ago
  I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data: > What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go? And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
  - 27183 4mo ago
    > And this is something LLMs can clearly do. ... > It knew that this is a dangerous thing I should not do in real life From the ChatGPT response you linked, all I see for sure is some matches on the following patterns: drop $thing from skyscraper --> bad behavior drop $thing --> physics can of $stuff --> contents in/out of can Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life". It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question? It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint. I believe you are projecting something that is not there onto a completely mindless stochastic process. [another edit] I found this helpful, you may also: https://www.0xkato.xyz/how-llms-actually-work/ https://www.0xkato.xyz/how-llms-actually-work/
beepbooptheory 4mo ago
But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it. It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
- hedgehog 4mo ago
  The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.

Diogenesian 4mo ago

This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."

throwaway7356 4mo ago
> yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability Where can I access such a Lisp expert system? If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.

PaulDavisThe1st 4mo ago

> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".

naasking 4mo ago
I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
- PaulDavisThe1st 4mo ago
  I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
  - naasking 4mo ago
    I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc. I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.
    - PaulDavisThe1st 4mo ago
      I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often. The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way. I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
      - naasking 4mo ago
        > The child knows how to catch the ball, without understanding. I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way. > I don't think that it is useful to say that LLMs understand anything they say, or that we say to them. This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.

andy99 4mo ago

I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns. There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.

emtel 4mo ago
> should I walk or drive to the car wash there is an entire genre of riddles based on this kind of misdirection that works on humans, such as "A plane crashed on the border or US and Canada. Where do they bury the survivors?" As with humans, comprehension is not a binary property of the agent - it is a quality that can be present in some situations and absent in others. LLMs may emit correct outputs sometimes because they do comprehend the input, and emit incorrect outputs in other cases when they do not comprehend the input. In order to show that LLMs can't comprehend, we'd have to show that there are no (or at least very few) situations in which they exhibit comprehension, not show that there are some situations in which they don't.

27183 4mo ago

> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. Is it possible you're making the following error described in the article? > The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people. Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation? I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.

SpicyLemonZest 4mo ago
I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data: > What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go? And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
- 27183 4mo ago
  > And this is something LLMs can clearly do. ... > It knew that this is a dangerous thing I should not do in real life From the ChatGPT response you linked, all I see for sure is some matches on the following patterns: drop $thing from skyscraper --> bad behavior drop $thing --> physics can of $stuff --> contents in/out of can Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life". It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question? It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint. I believe you are projecting something that is not there onto a completely mindless stochastic process. [another edit] I found this helpful, you may also: https://www.0xkato.xyz/how-llms-actually-work/ https://www.0xkato.xyz/how-llms-actually-work/

beepbooptheory 4mo ago

But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it. It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."

hedgehog 4mo ago
The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.