5 ms·

No LLMs don't think like people, they're architecturally incapable of doing so. They have, physically unlike humans no access to their own internal state and th

No LLMs don't think like people, they're architecturally incapable of doing so. They have, physically unlike humans no access to their own internal state and they're, save for a small context window, static systems. They also have no insights. There's a hilarious video about LLM Jailbreaks by Karpathy[1] from a week ago, where he shows how you can break model responses by asking the same question with a base64 string, preceding the prompt with an image of a panda(???) or just random word salad. LLM's are basically a validation of Searle's Chinese room. What they've proven is that you can build functioning systems that perform intelligent tasks purely at the level of syntax. But there is no (or very little) understanding of semantics. If I ask a person on how to end the world, whether I ask in French or English or base64 or perform a 50 word incantation beforehand likely does not matter. (unless of course the human is also just parroting an answer) [1] https://youtu.be/zjkBMFhNj_g?t=2974 https://youtu.be/zjkBMFhNj_g?t=2974

mcguire 3y ago
I was right there with you until you mentioned Searle. :-) The Chinese room argument is bad in that it hides an assumption of mind/body dualism. If you believe that humans have "souls" and other things do not, then you have a qualitative difference between a human or a machine. On the other hand, if you are a materialist then you are faced with the problem that humans don't have much understanding of semantics either. We're all chemical processes and it's hard for those to get much into semantics. But then, the difference between LLMs and humans becomes quantitative, sort of, and since I cannot say that LLMs and humans are qualitatively different, the only argument I can find is that in my experience, LLMs have never responded in a way that leads me to believe that they are anything other than a statistical model of language. Humans, on the other hand, are not a statistical model of language.
- Barrin92 3y ago
  Searle's one of the most die-hard materialist philosophers of mind around. It's his materialism that leads him to make his argument. Computers and human brains are both made out of atoms but that doesn't mean they're not qualitatively different. By that logic I"d be no different from a tree. There's qualitative differences between computers and human brains. Our cognition is biochemical and horribly slow, just by virtue of speed we are not working like LLMs. We're not doing tensor math in our heads, we don't have access to terrabytes of unaltered, digital data. It's because our bandwith and monkey brains are so slow that we're forced to operate at a level of semantics. We can't just make inferences from almost infinite amounts of data the same way we can't play chess like Stockfish or do math like a calculator. The dualism is precisely in the opposite view, that computation is somehow "substrate independent". Searle argues we can have AI that has understanding the way we do, just that it's going to look more like an organic brain as a result. The important insight from LLMs is that they're not like us at all but that doesn't make them less effective or intelligent. We do have plenty of understanding, we need to because we rely on a particular kind of reasoning, but artificial systems don't need to converge on that.
  - int_19h 3y ago
    Computation is a purely physical phenomenon, so no, saying that sentience is computation that is substrate-independent is not dualism - it's hardline materialism. Dualism is saying that sentience cannot be entirely reduced down to physical phenomena.
  - calf 3y ago
    LLMs nevertheless contain rudimentary theories, don't they? Like the Othello example demonstrating a spatial model that is emergent. LLMs are fast like calculators but it seems the optimization process that generates the parameter weights still produces "fuzzy semantics" and the Othello emergence is just one example.
  - mcguire 3y ago
    The Chinese Room gag asserts that, although the room behaves in every way like it is an intelligent Chinese speaker, we can see inside the room and determine that there is nothing there that intelligently understands Chinese. Searle seems to see a distinction between the "substrates" (which means an LLM cannot be intelligent; it's running on regular computer hardware and there's nothing to be found which "understands"), but unless someone can point out exactly what part of the substrate is intelligent, I am going to continue to suggest that his substrate difference is exactly identical to having a soul or not having a soul. I, as a materialist, assert that, if you dug through my noggin down to the level of subatomic particles you will never find anything that is intelligent or which understands. (Quantum mechanics does not help here, by the way. It just replaces a (rather naive) determinism with randomness---see Bell's Inequality.) There is no magic going on there. You and your tree are both doing nothing but chemistry and physics (you share something like 50% of your genes, by the way). And that means that a computer could, in principle, behave as intelligently as I do. That's "substrate independence". Now, whether a given system does do so or not is another question entirely. :-)
pixl97 3y ago
While you right about LLMs, you're not really making the case for humans well at all. Human insight is really easy to break, confidence men wouldn't really be a thing if it were hard to break. Simply putting a statement like "I love you" in front of a statement commonly overrides our intellect. Or offering a chocolate bar in trade of our passwords. If you want a human to tell you how to end the world, you'd just convince them to be your friend first.
int_19h 3y ago
Searle's Chinese room is a good example of begging the question. As for the rest of it, the LLM is basically "raw compute". You need a self-referential loop and long-term memories for it to even have the notion of self. But looking at it at that level and discounting it as "incapable of thinking" is missing the point - it's the larger system of which LLM is one part, albeit a key one (and which we're still trying to figure out how to build) that might actually be conscious etc.
TeMPOraL 3y ago
> What they've proven is that you can build functioning systems that perform intelligent tasks purely at the level of syntax. But there is no (or very little) understanding of semantics. That's not how LLMs work though, and I'm increasingly convinced that "syntax" and "semantics" are turning into annoyingly useless ideas people forget are descriptive, in the same way grammar books and dictionaries are descriptive. My model of LLMs is that, in training, they're positioning inputs in an absurdly high-dimensional[0] latent space. That space is big enough to encode anything you'd consider "syntax" and "semantics" on some subset of dimensions. As a result, the model sidesteps the issue of "source of meaning" - there is no meaning but that formed through associations (proximity). This is pretty much how we do it, too - when you think of "chair", there is no token for platonic ideal of a chair in your mind. The word "chair" has meanings and connotations defined via other words, which themselves are defined via other words, ad infinitum, with the only grounding being associations to sensory inputs. -- [0] - On the order of 100 000 dimensions for GPT-4, perhaps more now for GPT-4V / GPT-4-Turbo.
- somewhereoutth 3y ago
  > big enough Not even slightly. See Cantor's diagonal argument. See also Plato's cave for syntax vs semantics.
  - TeMPOraL 3y ago
    > Not even slightly. See Cantor's diagonal argument. Cantor's diagonal argument, if important, can just become one of the hundred thousand dimensions of the latent space. "Coordinate #12321 - degree of similarity to diagonal argument". > See also Plato's cave for syntax vs semantics. I don't see it. All there ever is are projections. There are just more dimensions in the mind space than in the physical space.

mcguire 3y ago

I was right there with you until you mentioned Searle. :-) The Chinese room argument is bad in that it hides an assumption of mind/body dualism. If you believe that humans have "souls" and other things do not, then you have a qualitative difference between a human or a machine. On the other hand, if you are a materialist then you are faced with the problem that humans don't have much understanding of semantics either. We're all chemical processes and it's hard for those to get much into semantics. But then, the difference between LLMs and humans becomes quantitative, sort of, and since I cannot say that LLMs and humans are qualitatively different, the only argument I can find is that in my experience, LLMs have never responded in a way that leads me to believe that they are anything other than a statistical model of language. Humans, on the other hand, are not a statistical model of language.

Barrin92 3y ago
Searle's one of the most die-hard materialist philosophers of mind around. It's his materialism that leads him to make his argument. Computers and human brains are both made out of atoms but that doesn't mean they're not qualitatively different. By that logic I"d be no different from a tree. There's qualitative differences between computers and human brains. Our cognition is biochemical and horribly slow, just by virtue of speed we are not working like LLMs. We're not doing tensor math in our heads, we don't have access to terrabytes of unaltered, digital data. It's because our bandwith and monkey brains are so slow that we're forced to operate at a level of semantics. We can't just make inferences from almost infinite amounts of data the same way we can't play chess like Stockfish or do math like a calculator. The dualism is precisely in the opposite view, that computation is somehow "substrate independent". Searle argues we can have AI that has understanding the way we do, just that it's going to look more like an organic brain as a result. The important insight from LLMs is that they're not like us at all but that doesn't make them less effective or intelligent. We do have plenty of understanding, we need to because we rely on a particular kind of reasoning, but artificial systems don't need to converge on that.
- int_19h 3y ago
  Computation is a purely physical phenomenon, so no, saying that sentience is computation that is substrate-independent is not dualism - it's hardline materialism. Dualism is saying that sentience cannot be entirely reduced down to physical phenomena.
- calf 3y ago
  LLMs nevertheless contain rudimentary theories, don't they? Like the Othello example demonstrating a spatial model that is emergent. LLMs are fast like calculators but it seems the optimization process that generates the parameter weights still produces "fuzzy semantics" and the Othello emergence is just one example.
- mcguire 3y ago
  The Chinese Room gag asserts that, although the room behaves in every way like it is an intelligent Chinese speaker, we can see inside the room and determine that there is nothing there that intelligently understands Chinese. Searle seems to see a distinction between the "substrates" (which means an LLM cannot be intelligent; it's running on regular computer hardware and there's nothing to be found which "understands"), but unless someone can point out exactly what part of the substrate is intelligent, I am going to continue to suggest that his substrate difference is exactly identical to having a soul or not having a soul. I, as a materialist, assert that, if you dug through my noggin down to the level of subatomic particles you will never find anything that is intelligent or which understands. (Quantum mechanics does not help here, by the way. It just replaces a (rather naive) determinism with randomness---see Bell's Inequality.) There is no magic going on there. You and your tree are both doing nothing but chemistry and physics (you share something like 50% of your genes, by the way). And that means that a computer could, in principle, behave as intelligently as I do. That's "substrate independence". Now, whether a given system does do so or not is another question entirely. :-)

pixl97 3y ago

While you right about LLMs, you're not really making the case for humans well at all. Human insight is really easy to break, confidence men wouldn't really be a thing if it were hard to break. Simply putting a statement like "I love you" in front of a statement commonly overrides our intellect. Or offering a chocolate bar in trade of our passwords. If you want a human to tell you how to end the world, you'd just convince them to be your friend first.

int_19h 3y ago

Searle's Chinese room is a good example of begging the question. As for the rest of it, the LLM is basically "raw compute". You need a self-referential loop and long-term memories for it to even have the notion of self. But looking at it at that level and discounting it as "incapable of thinking" is missing the point - it's the larger system of which LLM is one part, albeit a key one (and which we're still trying to figure out how to build) that might actually be conscious etc.

TeMPOraL 3y ago

> What they've proven is that you can build functioning systems that perform intelligent tasks purely at the level of syntax. But there is no (or very little) understanding of semantics. That's not how LLMs work though, and I'm increasingly convinced that "syntax" and "semantics" are turning into annoyingly useless ideas people forget are descriptive, in the same way grammar books and dictionaries are descriptive. My model of LLMs is that, in training, they're positioning inputs in an absurdly high-dimensional[0] latent space. That space is big enough to encode anything you'd consider "syntax" and "semantics" on some subset of dimensions. As a result, the model sidesteps the issue of "source of meaning" - there is no meaning but that formed through associations (proximity). This is pretty much how we do it, too - when you think of "chair", there is no token for platonic ideal of a chair in your mind. The word "chair" has meanings and connotations defined via other words, which themselves are defined via other words, ad infinitum, with the only grounding being associations to sensory inputs. -- [0] - On the order of 100 000 dimensions for GPT-4, perhaps more now for GPT-4V / GPT-4-Turbo.

somewhereoutth 3y ago
> big enough Not even slightly. See Cantor's diagonal argument. See also Plato's cave for syntax vs semantics.
- TeMPOraL 3y ago
  > Not even slightly. See Cantor's diagonal argument. Cantor's diagonal argument, if important, can just become one of the hundred thousand dimensions of the latent space. "Coordinate #12321 - degree of similarity to diagonal argument". > See also Plato's cave for syntax vs semantics. I don't see it. All there ever is are projections. There are just more dimensions in the mind space than in the physical space.