9 ms·

Large Language Model algorithms do not reason. They are statistical text generators, whose results are defined by their training data set. This is why the pap

Large Language Model algorithms do not reason. They are statistical text generators, whose results are defined by their training data set. This is why the paper cited reads thusly: Despite extensive research into the reasoning capabilities of Large Language Models (LLMs), most studies have failed to rigorously differentiate between inductive and deductive reasoning ... There is no differentiation because what was sought is the existence of what does not. The authors then postulate: This raises an essential question: In LLM reasoning, which poses a greater challenge - deductive or inductive reasoning? There is no such thing as "LLM reasoning." Therefore, the greatest challenge is accepting this fact and/or that anthropomorphism is a real thing.

astrange 2y ago
They are not statistical text generators. They are black boxes wrapped in an interface part of which is a statistical text generator. You can write programs into transformer models and run them deterministically; they aren't "statistical".
- godelski 2y ago
  Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. It's also still statistical if it gives you the same answer 99999/100000 times. You can definitely tune these things to be much more certain about certain problems but it tends to decrease capabilities in others. I also don't like the term "black box." While they aren't transparent they aren't completely opaque either. A big reason I didn't like the term is that I feel it also encourages people to not research this more. While we don't completely know what's going on I see no reason we can't. Is there some proof they their calculations are an unbreakable encryption or is it just a hard problem. I'm pretty sure it's the latter, and I think it's good to differentiate "we're dumb" from "indeterminate"
  - taylorius 2y ago
    The issue with comprehending LLM's internal workings is just that they're so big. The operation of the network itself is deterministic - but it's output is a set of probabilities over the value of the next token in the stream. At that point you have to choose one, based on that probability distribution, and then start the process again. That choice is where the randomness comes in.
    - godelski 2y ago
      Just to clarify, are you disagreing with me? Correcting me? Adding context for others? I guess I'm missing the intent
      - taylorius 2y ago
        Just adding context, elaborating a little on the deterministic / statistical aspects of an LLM. I don't disagree with what you said, I'm afraid - so not much prospect for a back and forth :-)
  - astrange 2y ago
    > Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. I was thinking temperature=0, but what more do you need than that? (nb there is also some randomness from non-associative FP operations happening in different orders) > I also don't like the term "black box." While they aren't transparent they aren't completely opaque either. A white box is a subset of a black box here. The important part is that the sampler/text generation is wrapped around the transformer model and is not actually the same thing as it.
    - godelski 2y ago
      > I was thinking temperature=0, but what more do you need than that? Temperature isn't a thing exclusive to LLMs or ML. It's a general technique from statistics where you're modifying the variance of the distribution (also see truncation). And as you're aware, temperature zero isn't actually temperature zero. So say such a thing makes something non-statistical is the same as saying prime numbers don't exist because any prime p * 0 is 0. > A white box is a subset of a black box here No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. Complexity isn't what makes something a black box, it's our lack of understanding that does. Of course this tends to coinside, but there's plenty of extremely complex things we have a great understanding of, even if not to infinite precision. Please take your passion and use it to drive you to learn about these things deeper. It's actually the same advice I give to the LLM fanboys who think it's AGI. But trying to dispel that myth without correct information makes it harder. It muddies the waters. These topics are extremely complex and the issue with the ML field is itself a hypocritical claim that it is easy (scale is all you need while calling these machines black boxes). I mean we can talk about the elephant in the room but certainly Von Neumann has said enough.
      - astrange 2y ago
        > No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. So it's a subset, because "understanding something" is a refinement of the state of "not understanding something" - you can just pretend you don't know what's happening in that part of the system and it is now a black box.
- HarHarVeryFunny 2y ago
  LLMs are statistical next-word predictors, wrapped in a stochastic next-word generator. The next-word predictions output by the model itself are based on the patterns and statistics of the training set. The next word that is actually generated is typically a random selection per a top-k or top-p sampling scheme.
stavros 2y ago
I really dislike these non-sequitur arguments of "LLMs do not reason, because <detail on how they work>", as if a detail on how they work unequivocally proves the impossibility of reasoning.
- quantadev 2y ago
  I've noticed that on HackerNews about 80% of all debates or discussions where people disagree, boils down to a disagreement about the definition of a word.
  - theknocker 2y ago
    [dead]
  - fedeb95 2y ago
    feel free to extend to other contexts. It was basically Socrates' argument.
  - throwawaytemp29 2y ago
    That is certainly true, for some definition of 80%.
- slashdave 2y ago
  I see. Well, I claim my pet rock can think. What do you mean it doesn't have brain cells, what kind of argument is that?
  - fragmede 2y ago
    My pet silicon rock was processed by TSMC in a design by Nvidia, and does this trick called inference that very much does look like thinking. Who made yours?
    - Y_Y 2y ago
      I deem your rock to be disqualified on the basis of its performance clearly having been enhanced by doping.
gaganyaan 2y ago
Define reason before making grandiose claims.
- quantadev 2y ago
  I think "reasoning" is the best word in the English language that I know of to describe what LLMs are doing. Lots of people disagree because for about 2000 years that word always implied consciousness, and LLMs definitely don't have consciousness.
  - gaganyaan 2y ago
    Define consciousness before making that claim either.
    - quantadev 2y ago
      I'm fine with the standard dictionary definition in this case. :P
      - gaganyaan 2y ago
        There's as many definitions as there are dictionaries, but let's try picking "awareness". LLMs are aware of many things, so they are conscious of those things, meaning they display some level of consciousness.
        quantadev 2y ago
        Nice one. Now do "qualia"
        gaganyaan 2y ago
        Qualia is a silly god of the gaps style argument. There's no difference between "seeing red" and "having your neurons tickled so that you think you're seeing red". People really don't understand consciousness, but it's actually quite simple when you realize that the question isn't "is this conscious?", it's "what is this conscious of?". A rock is not conscious of anything, because it has no state that represents anything. A sunflower is conscious of the sun's position, because it has state that represents that. A cat is conscious of much more, such as "this mouse is food". An LLM is conscious of much more, but less than a human. The LLM is a model that is conscious of many things, as experienced through the medium of human text.
aurareturn 2y ago
Brains do not reason - they are a neural network whose results are defined by their experiences, memories, and sensory inputs. I'm tired of reading comments from people who keep repeating that LLMs don't think, don't reason, isn't intelligence because it is not human. If your definition of the above is because it's not human, it's quite useless as a comment. We know LLMs aren't biological human brains. So what? Define what reasoning is to you. Then tell us why LLMs don't reason and why it matters.
- cscurmudgeon 2y ago
  Not OP. But: 1. Reasoning is ability to at least carry out proofs in FOL (first-order logic). FOL can simulate Turing Mach 2. LLMs are formally equivalent to only a subset of FOL. Why is this important? To model human mathematics, you need at least first-order logic. These arguments have been around for decades, e.g., Penrose. I am tired of people bringing up strawmen arguments ("Not intelligent because not human!")
  - logicchains 2y ago
    Transformers with chain of thought can use FOL: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923
    - Jweb_Guru 2y ago
      "Turing complete as long as you only use a polynomial number of steps in the size of the input" is another way of saying "not Turing complete" (doubly so when you look at the actual polynomials they're using). In fact, that paper just reaffirms that adding context doesn't improve formal expressive power all that much. And this is not equivalent to "well, Turing Machines are fake anyway because of finite space"--I can pretty trivially write programs in Coq, a total and terminating language, that will take far longer than the age of the universe to normalize but can be described in very little input.
  - scarmig 2y ago
    Is there a proof that collections of human neurons are capable of carrying out proofs in first order logic, in full generality?
godelski 2y ago
> They are statistical text generators, whose results are defined by their training data set Honestly I'm pissed at the research community. It's fraud. If you don't know what's in the training data you simply cannot differentiate reasoning from memorization. Beyond that, the experiments are almost certainly in the training data. Like come on! I feel like I'm going crazy here. How can any coder not think LLMs are training on oct and hex‽ https://news.ycombinator.com/item?id=41422751 https://news.ycombinator.com/item?id=41422751
- famouswaffles 2y ago
  Of course they've been trained on oct and hex. The question is would the results be largely valid if this was done on a human who had learnt how to perform base 8, 9, 11 etc arithmetic instead ? I mean, they're clearly not trying to test the ability to derive base arithmetic from scratch.
  - deleted 2y ago
    [deleted]
anon373839 2y ago
> Large Language Model algorithms do not reason. My belief is that what LLMs do is best described as approximating the outputs of reasoning. This is different from reasoning itself and also different from simply regurgitating the most similar training examples. I also think it's extremely challenging for people to understand the significance of LLM output because language is intertwined with meaning. So if you construct a valid and seemingly responsive sequence of words in response to a prompt, there is a strong illusion of purpose behind those words, even if they were assembled in a completely naive fashion.
- levitatorius 2y ago
  This! One simple argument is that language is NOT a magical reasoning substance in itself, but a communication medium. It is medium for passing (a) meaning. So first there is a meaningful thought (worth of sharing), then an agent puts a SIGNIFIER on that meaningful thought, then communicates it to the recipient. Communication medium can be a sentence, it can also be an eyewink or a tail wiggle. Or a whistle. The "language" can be created on the spot, if two subjects get a meaning of signifier by intuition (e.g. I look at the object, you follow my gaze). So the fallacy of the whole LLM field is the belief that language has some intrinsic meaning. Or if you mix the artifacts of language in some very smart way, the meaning will emerge. But it doesn't work if meaning occurs before the word. The text in books has no reasoning, it was authors. The machine shuffling the text fragments does not have a meaningful thought. The engineer which devised a shuffling machine had some meaningful thought, the users of the machine have same thoughts, but not the machine itself. To put it another way, if there was an artificial system capable of producing meaningful thoughts, it is not a presence of language which produces a proof, it's communication. Communication requires an agent (as in "agency") and an intent. We have neither in LLM. As to the argument that we ourselves are mere stochastic parrots - of course we can produce word salads, or fake mimics of coherent text, it is not a proof that LLM IS the way our minds work. It is just a witness to the fact language is a flexible medium for the meanings behind - it can just as well be used for cheating, pretending, etc.
  - mistermann 2y ago
    > So the fallacy of... The text from this point on seems to have lost contextual awareness of what preceded it (which was excellent imho).
seanmcdirmid 2y ago
LLMs are statistical text generators whose results depend on the model and the context given. They have gotten so good because the context they can effectively operate over keeps getting really big. If you take the model and operate on a small context, you will get very uninteresting results. The only reason it seems like it is reasoning is because it’s probably stuffing a lot of reasoning in its context, and regurgitating that out in ways that are statically weighted with other things in the context on what is being reasoned about. Frankly, even most commenters on HN don’t get how LLMs operate, thinking the model itself is what knows about different bases like hex and oct, when really, it searched up a bunch of material on different bases to include in the context before the model was ever activated.

astrange 2y ago

They are not statistical text generators. They are black boxes wrapped in an interface part of which is a statistical text generator. You can write programs into transformer models and run them deterministically; they aren't "statistical".

godelski 2y ago
Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. It's also still statistical if it gives you the same answer 99999/100000 times. You can definitely tune these things to be much more certain about certain problems but it tends to decrease capabilities in others. I also don't like the term "black box." While they aren't transparent they aren't completely opaque either. A big reason I didn't like the term is that I feel it also encourages people to not research this more. While we don't completely know what's going on I see no reason we can't. Is there some proof they their calculations are an unbreakable encryption or is it just a hard problem. I'm pretty sure it's the latter, and I think it's good to differentiate "we're dumb" from "indeterminate"
- taylorius 2y ago
  The issue with comprehending LLM's internal workings is just that they're so big. The operation of the network itself is deterministic - but it's output is a set of probabilities over the value of the next token in the stream. At that point you have to choose one, based on that probability distribution, and then start the process again. That choice is where the randomness comes in.
  - godelski 2y ago
    Just to clarify, are you disagreing with me? Correcting me? Adding context for others? I guess I'm missing the intent
    - taylorius 2y ago
      Just adding context, elaborating a little on the deterministic / statistical aspects of an LLM. I don't disagree with what you said, I'm afraid - so not much prospect for a back and forth :-)
- astrange 2y ago
  > Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. I was thinking temperature=0, but what more do you need than that? (nb there is also some randomness from non-associative FP operations happening in different orders) > I also don't like the term "black box." While they aren't transparent they aren't completely opaque either. A white box is a subset of a black box here. The important part is that the sampler/text generation is wrapped around the transformer model and is not actually the same thing as it.
  - godelski 2y ago
    > I was thinking temperature=0, but what more do you need than that? Temperature isn't a thing exclusive to LLMs or ML. It's a general technique from statistics where you're modifying the variance of the distribution (also see truncation). And as you're aware, temperature zero isn't actually temperature zero. So say such a thing makes something non-statistical is the same as saying prime numbers don't exist because any prime p * 0 is 0. > A white box is a subset of a black box here No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. Complexity isn't what makes something a black box, it's our lack of understanding that does. Of course this tends to coinside, but there's plenty of extremely complex things we have a great understanding of, even if not to infinite precision. Please take your passion and use it to drive you to learn about these things deeper. It's actually the same advice I give to the LLM fanboys who think it's AGI. But trying to dispel that myth without correct information makes it harder. It muddies the waters. These topics are extremely complex and the issue with the ML field is itself a hypocritical claim that it is easy (scale is all you need while calling these machines black boxes). I mean we can talk about the elephant in the room but certainly Von Neumann has said enough.
    - astrange 2y ago
      > No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. So it's a subset, because "understanding something" is a refinement of the state of "not understanding something" - you can just pretend you don't know what's happening in that part of the system and it is now a black box.
HarHarVeryFunny 2y ago
LLMs are statistical next-word predictors, wrapped in a stochastic next-word generator. The next-word predictions output by the model itself are based on the patterns and statistics of the training set. The next word that is actually generated is typically a random selection per a top-k or top-p sampling scheme.

stavros 2y ago

I really dislike these non-sequitur arguments of "LLMs do not reason, because <detail on how they work>", as if a detail on how they work unequivocally proves the impossibility of reasoning.

quantadev 2y ago
I've noticed that on HackerNews about 80% of all debates or discussions where people disagree, boils down to a disagreement about the definition of a word.
- theknocker 2y ago
  [dead]
- fedeb95 2y ago
  feel free to extend to other contexts. It was basically Socrates' argument.
- throwawaytemp29 2y ago
  That is certainly true, for some definition of 80%.
slashdave 2y ago
I see. Well, I claim my pet rock can think. What do you mean it doesn't have brain cells, what kind of argument is that?
- fragmede 2y ago
  My pet silicon rock was processed by TSMC in a design by Nvidia, and does this trick called inference that very much does look like thinking. Who made yours?
  - Y_Y 2y ago
    I deem your rock to be disqualified on the basis of its performance clearly having been enhanced by doping.

gaganyaan 2y ago

Define reason before making grandiose claims.

quantadev 2y ago
I think "reasoning" is the best word in the English language that I know of to describe what LLMs are doing. Lots of people disagree because for about 2000 years that word always implied consciousness, and LLMs definitely don't have consciousness.
- gaganyaan 2y ago
  Define consciousness before making that claim either.
  - quantadev 2y ago
    I'm fine with the standard dictionary definition in this case. :P
    - gaganyaan 2y ago
      There's as many definitions as there are dictionaries, but let's try picking "awareness". LLMs are aware of many things, so they are conscious of those things, meaning they display some level of consciousness.
      - quantadev 2y ago
        Nice one. Now do "qualia"
        gaganyaan 2y ago
        Qualia is a silly god of the gaps style argument. There's no difference between "seeing red" and "having your neurons tickled so that you think you're seeing red". People really don't understand consciousness, but it's actually quite simple when you realize that the question isn't "is this conscious?", it's "what is this conscious of?". A rock is not conscious of anything, because it has no state that represents anything. A sunflower is conscious of the sun's position, because it has state that represents that. A cat is conscious of much more, such as "this mouse is food". An LLM is conscious of much more, but less than a human. The LLM is a model that is conscious of many things, as experienced through the medium of human text.

aurareturn 2y ago

Brains do not reason - they are a neural network whose results are defined by their experiences, memories, and sensory inputs. I'm tired of reading comments from people who keep repeating that LLMs don't think, don't reason, isn't intelligence because it is not human. If your definition of the above is because it's not human, it's quite useless as a comment. We know LLMs aren't biological human brains. So what? Define what reasoning is to you. Then tell us why LLMs don't reason and why it matters.

cscurmudgeon 2y ago
Not OP. But: 1. Reasoning is ability to at least carry out proofs in FOL (first-order logic). FOL can simulate Turing Mach 2. LLMs are formally equivalent to only a subset of FOL. Why is this important? To model human mathematics, you need at least first-order logic. These arguments have been around for decades, e.g., Penrose. I am tired of people bringing up strawmen arguments ("Not intelligent because not human!")
- logicchains 2y ago
  Transformers with chain of thought can use FOL: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923
  - Jweb_Guru 2y ago
    "Turing complete as long as you only use a polynomial number of steps in the size of the input" is another way of saying "not Turing complete" (doubly so when you look at the actual polynomials they're using). In fact, that paper just reaffirms that adding context doesn't improve formal expressive power all that much. And this is not equivalent to "well, Turing Machines are fake anyway because of finite space"--I can pretty trivially write programs in Coq, a total and terminating language, that will take far longer than the age of the universe to normalize but can be described in very little input.
- scarmig 2y ago
  Is there a proof that collections of human neurons are capable of carrying out proofs in first order logic, in full generality?

godelski 2y ago

> They are statistical text generators, whose results are defined by their training data set Honestly I'm pissed at the research community. It's fraud. If you don't know what's in the training data you simply cannot differentiate reasoning from memorization. Beyond that, the experiments are almost certainly in the training data. Like come on! I feel like I'm going crazy here. How can any coder not think LLMs are training on oct and hex‽ https://news.ycombinator.com/item?id=41422751 https://news.ycombinator.com/item?id=41422751

famouswaffles 2y ago
Of course they've been trained on oct and hex. The question is would the results be largely valid if this was done on a human who had learnt how to perform base 8, 9, 11 etc arithmetic instead ? I mean, they're clearly not trying to test the ability to derive base arithmetic from scratch.
- deleted 2y ago
  [deleted]

anon373839 2y ago

> Large Language Model algorithms do not reason. My belief is that what LLMs do is best described as approximating the outputs of reasoning. This is different from reasoning itself and also different from simply regurgitating the most similar training examples. I also think it's extremely challenging for people to understand the significance of LLM output because language is intertwined with meaning. So if you construct a valid and seemingly responsive sequence of words in response to a prompt, there is a strong illusion of purpose behind those words, even if they were assembled in a completely naive fashion.

levitatorius 2y ago
This! One simple argument is that language is NOT a magical reasoning substance in itself, but a communication medium. It is medium for passing (a) meaning. So first there is a meaningful thought (worth of sharing), then an agent puts a SIGNIFIER on that meaningful thought, then communicates it to the recipient. Communication medium can be a sentence, it can also be an eyewink or a tail wiggle. Or a whistle. The "language" can be created on the spot, if two subjects get a meaning of signifier by intuition (e.g. I look at the object, you follow my gaze). So the fallacy of the whole LLM field is the belief that language has some intrinsic meaning. Or if you mix the artifacts of language in some very smart way, the meaning will emerge. But it doesn't work if meaning occurs before the word. The text in books has no reasoning, it was authors. The machine shuffling the text fragments does not have a meaningful thought. The engineer which devised a shuffling machine had some meaningful thought, the users of the machine have same thoughts, but not the machine itself. To put it another way, if there was an artificial system capable of producing meaningful thoughts, it is not a presence of language which produces a proof, it's communication. Communication requires an agent (as in "agency") and an intent. We have neither in LLM. As to the argument that we ourselves are mere stochastic parrots - of course we can produce word salads, or fake mimics of coherent text, it is not a proof that LLM IS the way our minds work. It is just a witness to the fact language is a flexible medium for the meanings behind - it can just as well be used for cheating, pretending, etc.
- mistermann 2y ago
  > So the fallacy of... The text from this point on seems to have lost contextual awareness of what preceded it (which was excellent imho).

seanmcdirmid 2y ago

LLMs are statistical text generators whose results depend on the model and the context given. They have gotten so good because the context they can effectively operate over keeps getting really big. If you take the model and operate on a small context, you will get very uninteresting results. The only reason it seems like it is reasoning is because it’s probably stuffing a lot of reasoning in its context, and regurgitating that out in ways that are statically weighted with other things in the context on what is being reasoned about. Frankly, even most commenters on HN don’t get how LLMs operate, thinking the model itself is what knows about different bases like hex and oct, when really, it searched up a bunch of material on different bases to include in the context before the model was ever activated.