6 ms·

Conclusion is obvious, but the paper is still probably necessary. Of course LLM's can't reason. They pattern match answers to previously asked questions, and h

by tovej 3y ago

Conclusion is obvious, but the paper is still probably necessary. Of course LLM's can't reason. They pattern match answers to previously asked questions, and humans will read the text as a reasonable answer because we assign meaning to it, but there is simply no way an LLM could use a "mental model" to "reason" about a problem other than constructing sentences out of probable matches it's been trained on.

mr_toad 3y ago
> Of course LLM's can't reason. They pattern match You’re assuming that reasoning isn’t just pattern matching.
sfn42 3y ago
Love how the ai hypebros always downvote answers like this. Everyone's just insisting that LLMs are AGI just because they kinda seem that way.
- viraptor 3y ago
  Ftr, I mostly agree with the comment but downvoted it for the low quality "of course that's the answer, obviously" padded with some unhelpful tropes about trained pattern matching. (Let us know when you can prove that human communication is not just trained pattern matching)
  - tovej 3y ago
    Ok, I'll bite. I personally don't think human communication is simply pattern matching (e.g. we have drives, motives, relationships that we want to tell each other, these are internal systems, outside the visible band of communication). That's a little off topic though, my post was not about communication, but reasoning. Pattern matching in LLMs is specifically at the language level, as in "predict the next token in a stream", I should have probably specified this bit. You seem to refer to a more general kind of pattern matching when you say, where we're looking at a complicated structural model within which humans reason about the behavior of different system components and their interactions. There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. If all you have is language, and predictive text, there is no way to use this to simulate and predict the rules in a higher-dimensional space than "stream of tokens", or let's put it another way: there would need to be an extremely high correlation between the pattern in the problem space and in language space for the likelihood of the model guessing right to be anywhere near human levels.
    - stevenhuang 3y ago
      > There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. You claim this as if it is a foregone conclusion yet it is anything but. This is an active area of study, even within human cognition https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding A complex system clearly does not need to be designed to evoke some behavior. Just look to the concept of emergent phenomenon. See evolution through natural selection as an obvious counter example. Evolution selects for fitness and not reasoning ability. Yet here we are. It is not known if the same isn't happening now for the transformer model. LLMs are universal function approximators. If reasoning ability is at all representable through such a process, it is possible for LLMs to find it. https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
      - tovej 3y ago
        Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the stream tokens in an LLM. This means an LLM can do arbitrary language manipulation. If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it. Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back. This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems. Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function. If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.
  - sfn42 3y ago
    I don't need to prove anything. We know for a fact that LLMs are stochastic parrots. That's why they confabulate, if they were "intelligent" they wouldn't just pass off nonsense as fact. They have been fed incredible amounts of data which allows them to stochastically parrot really complex things, this makes people think they're intelligent. Designing a test that actually tests intelligence rather than parroting ability is tricky, but whenever people do they fail them. I just logged into ChatGPT and asked it repeatedly to generate a novel thought experiment intended to prove that LLMs are incapable of reasoning. I kept trying to refine it by explaining what was wrong with each suggestion. It completely failed. It just spat out variations of well known dilemmas or paradoxes, even when I specifically asked it to create something original it just made nonsense.
    - mcswell 3y ago
      "whenever people do they fail them": I assume the antecedent of "they" is LLMs, not people.
- danielbln 3y ago
  You don't have to be an AI "hype bro" to take issue with the reductive and trite notion that LLM's are stochastic parrots only. There is a continuum between that and AGI.
  - PaulDavisThe1st 3y ago
    A continuum like the ape climbing to the top of a tree being part of a continuum that includes landing on the moon?
    - danielbln 3y ago
      Pretty much, I'd say we're way past the fleecing and poop flinging at this point and well into persistence hunting, maybe even control of fire?
    - int_19h 3y ago
      I'd say it's more like the ape making and using its first crude tool by itself.
  - sfn42 3y ago
    Well I take issue with the reductive and trite notion that just because an LLM can generate plausible text it's suddenly maybe conscious and intelligent, maybe about to end humanity etc. It's exactly like the crypto hype wave. Everyone dreaming up semi-plausible futures, based on a whole chain of unfounded assumptions.
    - pixl97 3y ago
      The only thing the LLM is missing is an self actuation loop. We put a camera on a multimodal LLM, it interprets the visual world and sees before it a bunch of blocks. It looks at the task list it has that says "pick up red blocks, put them in blue bin". The visual components identifies the red blocks and the textual component issues commands to its drive unit which calculates the best path and how to use it's manipulators to pick up the blocks. This is a very basic chain of existence. We have world identification, we have actuation on motivation, we have interaction with the environment. We can do this now. These goals have already been achieved. Companies are already testing more complex models with much more general instructions such as "Pick up everything you think is trash" or "Organize this room" to see the emergent behaviors that come out of the models. You are seemingly a few years behind what has already been done, and why people are starting to get concerned.
- dodslaser 3y ago
  I think it's pretty funny that the moment we taught a computer to talk like a human people immediately assumed it was sentient, let alone intelligent.
  - ben_w 3y ago
    And this assumption has only gotten more common in the ~56 years since (so the anecdote goes) ELIZA fooled Weizenbaum's secretary. "Consciousness" has ~22 different definitions; some are easily met, some are not even met by the humans giving those definitions (for example, I've seen people here on HN who write as if consciousness can somehow magically solve the halting problem). There's only one test for it in AI, that I've heard of, and that I think is worth exploring: https://www.youtube.com/watch?v=LWf3szP6L80 https://www.youtube.com/watch?v=LWf3szP6L80
  - jofla_net 3y ago
    I suspect this has to do with how shallowly people even regard their fellow man. To the point that if the outputs of an entity parrot humans well enough then, well, there couldn't be any differences whatsoever, and it must bear all the titles which we gift to humans.
  - int_19h 3y ago
    It has to do with the fact that all our definitions of "sentient" and "intelligent" are extremely vague, often vacuous, and generally boil down to "I know it when I see it" in practice. So there should be absolutely no surprise that, when something comes along that behaves in certain ways very much like a human (and unlike anything else), it is assumed to share the basic qualities of such. It is how we judge other people, after all - it's not like you do a brain scan to determine whether someone you are talking to is "sentient", whatever that means to you. You judge that by their behavior, modelling what a "sentient" would do and comparing the result to what they actually do.
- jstummbillig 3y ago
  I think both of your comments are just boring and condescending.
koe123 3y ago
On the other hand, can we conclusively say that humans aren’t really advanced biological stochastic parrots?
naasking 3y ago
> Of course LLM's can't reason That they are not effective at some forms of reasoning does not entail they can't reason.
mcswell 3y ago
The conclusion may be obvious to you and me (although it's hard to know for certain, since these available LLMs are black boxes). But it's definitely not obvious to everyone. There are plenty of people saying this is the dawn of AGI, or that we're a few short steps from AGI. Whereas people like Gary Marcus (who knows tons more than I do) says LLMs are going off in the wrong direction.
- visarga 3y ago
  Yes, LLMs can't reason 100% correctly, but neither do humans. We can often reason correctly, but it's not always the case. Even reasoning, fundamental as it is, comes from feedback. Feedback from our actions teaches us how to reason. Learning from feedback is more general than reasoning - AI agents can definitely learn this way too, if they have enough freedom to explore. But you can't do it with supervised training sets alone. You need to put language models into agents in environments, give them goals and rewards. Then they can make their own training data and mistakes, build up their own experience. Can't teach an AI based on how people make mistakes, it needs to fix its own mistakes, but that means to deploy it in the wild, where there are consequences for errors. If you remember, DeepMind first tried to train a Go model on human game play, but it was limited. Then they started from scratch, and learning from feedback alone they surpassed human level, even though the feedback was one single bit of information at the end of a whole self-play game. And it had no pre-training prior to learning Go, unlike human players.
  - somewhereoutth 3y ago
    That presupposes that language encodes the world we live in completely, whereas in fact language is meaningless without the shared context of reality. 'up' and 'down' are only meaningful to intelligence that can experience space. Essentially LLMs just are oracles for the shadows on Plato's cave.
    - visarga 3y ago
      Do a reverse Chinese room experiment - remove from a human all the abilities multi-modal LLMs gain after training on human media. What's left? Homo ferus. Most of our intelligence is encoded in the environment and language, it's a collective process, not an individual one. We're collectively, not individually, very smart.

by tovej 3y ago

mr_toad 3y ago

> Of course LLM's can't reason. They pattern match You’re assuming that reasoning isn’t just pattern matching.

sfn42 3y ago

Love how the ai hypebros always downvote answers like this. Everyone's just insisting that LLMs are AGI just because they kinda seem that way.

viraptor 3y ago
Ftr, I mostly agree with the comment but downvoted it for the low quality "of course that's the answer, obviously" padded with some unhelpful tropes about trained pattern matching. (Let us know when you can prove that human communication is not just trained pattern matching)
- tovej 3y ago
  Ok, I'll bite. I personally don't think human communication is simply pattern matching (e.g. we have drives, motives, relationships that we want to tell each other, these are internal systems, outside the visible band of communication). That's a little off topic though, my post was not about communication, but reasoning. Pattern matching in LLMs is specifically at the language level, as in "predict the next token in a stream", I should have probably specified this bit. You seem to refer to a more general kind of pattern matching when you say, where we're looking at a complicated structural model within which humans reason about the behavior of different system components and their interactions. There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. If all you have is language, and predictive text, there is no way to use this to simulate and predict the rules in a higher-dimensional space than "stream of tokens", or let's put it another way: there would need to be an extremely high correlation between the pattern in the problem space and in language space for the likelihood of the model guessing right to be anywhere near human levels.
  - stevenhuang 3y ago
    > There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. You claim this as if it is a foregone conclusion yet it is anything but. This is an active area of study, even within human cognition https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding A complex system clearly does not need to be designed to evoke some behavior. Just look to the concept of emergent phenomenon. See evolution through natural selection as an obvious counter example. Evolution selects for fitness and not reasoning ability. Yet here we are. It is not known if the same isn't happening now for the transformer model. LLMs are universal function approximators. If reasoning ability is at all representable through such a process, it is possible for LLMs to find it. https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
    - tovej 3y ago
      Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the stream tokens in an LLM. This means an LLM can do arbitrary language manipulation. If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it. Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back. This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems. Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function. If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.
- sfn42 3y ago
  I don't need to prove anything. We know for a fact that LLMs are stochastic parrots. That's why they confabulate, if they were "intelligent" they wouldn't just pass off nonsense as fact. They have been fed incredible amounts of data which allows them to stochastically parrot really complex things, this makes people think they're intelligent. Designing a test that actually tests intelligence rather than parroting ability is tricky, but whenever people do they fail them. I just logged into ChatGPT and asked it repeatedly to generate a novel thought experiment intended to prove that LLMs are incapable of reasoning. I kept trying to refine it by explaining what was wrong with each suggestion. It completely failed. It just spat out variations of well known dilemmas or paradoxes, even when I specifically asked it to create something original it just made nonsense.
  - mcswell 3y ago
    "whenever people do they fail them": I assume the antecedent of "they" is LLMs, not people.
danielbln 3y ago
You don't have to be an AI "hype bro" to take issue with the reductive and trite notion that LLM's are stochastic parrots only. There is a continuum between that and AGI.
- PaulDavisThe1st 3y ago
  A continuum like the ape climbing to the top of a tree being part of a continuum that includes landing on the moon?
  - danielbln 3y ago
    Pretty much, I'd say we're way past the fleecing and poop flinging at this point and well into persistence hunting, maybe even control of fire?
  - int_19h 3y ago
    I'd say it's more like the ape making and using its first crude tool by itself.
- sfn42 3y ago
  Well I take issue with the reductive and trite notion that just because an LLM can generate plausible text it's suddenly maybe conscious and intelligent, maybe about to end humanity etc. It's exactly like the crypto hype wave. Everyone dreaming up semi-plausible futures, based on a whole chain of unfounded assumptions.
  - pixl97 3y ago
    The only thing the LLM is missing is an self actuation loop. We put a camera on a multimodal LLM, it interprets the visual world and sees before it a bunch of blocks. It looks at the task list it has that says "pick up red blocks, put them in blue bin". The visual components identifies the red blocks and the textual component issues commands to its drive unit which calculates the best path and how to use it's manipulators to pick up the blocks. This is a very basic chain of existence. We have world identification, we have actuation on motivation, we have interaction with the environment. We can do this now. These goals have already been achieved. Companies are already testing more complex models with much more general instructions such as "Pick up everything you think is trash" or "Organize this room" to see the emergent behaviors that come out of the models. You are seemingly a few years behind what has already been done, and why people are starting to get concerned.
dodslaser 3y ago
I think it's pretty funny that the moment we taught a computer to talk like a human people immediately assumed it was sentient, let alone intelligent.
- ben_w 3y ago
  And this assumption has only gotten more common in the ~56 years since (so the anecdote goes) ELIZA fooled Weizenbaum's secretary. "Consciousness" has ~22 different definitions; some are easily met, some are not even met by the humans giving those definitions (for example, I've seen people here on HN who write as if consciousness can somehow magically solve the halting problem). There's only one test for it in AI, that I've heard of, and that I think is worth exploring: https://www.youtube.com/watch?v=LWf3szP6L80 https://www.youtube.com/watch?v=LWf3szP6L80
- jofla_net 3y ago
  I suspect this has to do with how shallowly people even regard their fellow man. To the point that if the outputs of an entity parrot humans well enough then, well, there couldn't be any differences whatsoever, and it must bear all the titles which we gift to humans.
- int_19h 3y ago
  It has to do with the fact that all our definitions of "sentient" and "intelligent" are extremely vague, often vacuous, and generally boil down to "I know it when I see it" in practice. So there should be absolutely no surprise that, when something comes along that behaves in certain ways very much like a human (and unlike anything else), it is assumed to share the basic qualities of such. It is how we judge other people, after all - it's not like you do a brain scan to determine whether someone you are talking to is "sentient", whatever that means to you. You judge that by their behavior, modelling what a "sentient" would do and comparing the result to what they actually do.
jstummbillig 3y ago
I think both of your comments are just boring and condescending.

koe123 3y ago

On the other hand, can we conclusively say that humans aren’t really advanced biological stochastic parrots?

naasking 3y ago

> Of course LLM's can't reason That they are not effective at some forms of reasoning does not entail they can't reason.

mcswell 3y ago

The conclusion may be obvious to you and me (although it's hard to know for certain, since these available LLMs are black boxes). But it's definitely not obvious to everyone. There are plenty of people saying this is the dawn of AGI, or that we're a few short steps from AGI. Whereas people like Gary Marcus (who knows tons more than I do) says LLMs are going off in the wrong direction.

visarga 3y ago
Yes, LLMs can't reason 100% correctly, but neither do humans. We can often reason correctly, but it's not always the case. Even reasoning, fundamental as it is, comes from feedback. Feedback from our actions teaches us how to reason. Learning from feedback is more general than reasoning - AI agents can definitely learn this way too, if they have enough freedom to explore. But you can't do it with supervised training sets alone. You need to put language models into agents in environments, give them goals and rewards. Then they can make their own training data and mistakes, build up their own experience. Can't teach an AI based on how people make mistakes, it needs to fix its own mistakes, but that means to deploy it in the wild, where there are consequences for errors. If you remember, DeepMind first tried to train a Go model on human game play, but it was limited. Then they started from scratch, and learning from feedback alone they surpassed human level, even though the feedback was one single bit of information at the end of a whole self-play game. And it had no pre-training prior to learning Go, unlike human players.
- somewhereoutth 3y ago
  That presupposes that language encodes the world we live in completely, whereas in fact language is meaningless without the shared context of reality. 'up' and 'down' are only meaningful to intelligence that can experience space. Essentially LLMs just are oracles for the shadows on Plato's cave.
  - visarga 3y ago
    Do a reverse Chinese room experiment - remove from a human all the abilities multi-modal LLMs gain after training on human media. What's left? Homo ferus. Most of our intelligence is encoded in the environment and language, it's a collective process, not an individual one. We're collectively, not individually, very smart.