5 ms·

Because there is no "reasoning" right? LLMs create output that appear to have been reasoned about, but this is not at all reality

nathan_compton 2y ago
I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.
- onemoresoop 2y ago
  That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going
  - bckr 2y ago
    Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
    - swatcoder 2y ago
      Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
      - onemoresoop 2y ago
        > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
      - Ekaros 2y ago
        I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
  - JustBreath 2y ago
    Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.
cmrdporcupine 2y ago
I agree, but apologists will say that that's all the human brain is doing, too. Myself, I think that's a terrible false analogy, and also have a higher bar for anything to be called artificial "intelligence." These programs shouldn't be using personal pronouns or statements that imply cognition or esp reasoning. It's deceptive.
- AnimalMuppet 2y ago
  I think nathan_compton is right in describing LLMs as "a single pass, feed forward, process calculating outputs from inputs". That is not all the human brain is doing. Human reasoning is not single pass - far from it. So I concur that the apologists are wrong about what the human brain is doing, and therefore wrong about the similarity to LLMs.
  - Kim_Bruning 2y ago
    If an LLM gets extra passes, it can be smarter. This is especially important for eg. strict logic puzzles, where error correction is needed.
  - cmrdporcupine 2y ago
    My gut instinct / guess is that there's "stuff" in the brain that is a kind of system for doing the kind of thing that symbolic logic systems do: proposition management, unification, backtracking, etc. And I think this stuff is probably there from a very early age. Also, generally, while humans can be really bad at reasoning, when they're shown why they're wrong they're able to retrace their steps and fix -- assuming they're not emotionally attached to some outcome. I'm terrible at math, but if I slow down and have it explained, I'll get it (and then forget it.) LLMs... they don't work that way. Tell them how they're wrong or to explain how they got to some result... forget it, they can't retrace and fix.
    - Kim_Bruning 2y ago
      I posted several conversations with models in this thread. I did -in fact- manage to get them to retrace and fix where needed. GPT-4o was a lot of work. GPT-3 is fast but that was a long conversation catching all its errors (so not posted, but I can if you insist). 4 caught its error almost immediately. Claude-3.5 was on to us from the get-go and didn't need correcting.
wongarsu 2y ago
In principle you would expect reasoning to eventually emerge. During training LLMs predict large volumes of text, and that text contains a lot of arguments and chains of thought. You can achieve some success predicting these with pattern matching, but the best way to complete arguments at scale is a reasoning process. But I think we are only scratching the surface of achieving this. Based on my admittedly casual observation of current model capabilities you need a model that's fairly big (closer to 70B than 7B) and trained far beyond what's implied by the Chinchilla metric.
caesil 2y ago
Why not exactly? The model weights cannot encode a sort of math engine? The hidden state cannot encode carryover values? Why do we assume these things can't happen at some level?
- jasfi 2y ago
  I agree. The reasoning is there, and becoming more capable every year (across the various models). It's easy to look for limitations, but what was once glaring problems are now much more subtle.

nathan_compton 2y ago

I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.

onemoresoop 2y ago
That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going
- bckr 2y ago
  Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
  - swatcoder 2y ago
    Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
    - onemoresoop 2y ago
      > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
    - Ekaros 2y ago
      I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
- JustBreath 2y ago
  Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.

cmrdporcupine 2y ago

I agree, but apologists will say that that's all the human brain is doing, too. Myself, I think that's a terrible false analogy, and also have a higher bar for anything to be called artificial "intelligence." These programs shouldn't be using personal pronouns or statements that imply cognition or esp reasoning. It's deceptive.

AnimalMuppet 2y ago
I think nathan_compton is right in describing LLMs as "a single pass, feed forward, process calculating outputs from inputs". That is not all the human brain is doing. Human reasoning is not single pass - far from it. So I concur that the apologists are wrong about what the human brain is doing, and therefore wrong about the similarity to LLMs.
- Kim_Bruning 2y ago
  If an LLM gets extra passes, it can be smarter. This is especially important for eg. strict logic puzzles, where error correction is needed.
- cmrdporcupine 2y ago
  My gut instinct / guess is that there's "stuff" in the brain that is a kind of system for doing the kind of thing that symbolic logic systems do: proposition management, unification, backtracking, etc. And I think this stuff is probably there from a very early age. Also, generally, while humans can be really bad at reasoning, when they're shown why they're wrong they're able to retrace their steps and fix -- assuming they're not emotionally attached to some outcome. I'm terrible at math, but if I slow down and have it explained, I'll get it (and then forget it.) LLMs... they don't work that way. Tell them how they're wrong or to explain how they got to some result... forget it, they can't retrace and fix.
  - Kim_Bruning 2y ago
    I posted several conversations with models in this thread. I did -in fact- manage to get them to retrace and fix where needed. GPT-4o was a lot of work. GPT-3 is fast but that was a long conversation catching all its errors (so not posted, but I can if you insist). 4 caught its error almost immediately. Claude-3.5 was on to us from the get-go and didn't need correcting.

wongarsu 2y ago

In principle you would expect reasoning to eventually emerge. During training LLMs predict large volumes of text, and that text contains a lot of arguments and chains of thought. You can achieve some success predicting these with pattern matching, but the best way to complete arguments at scale is a reasoning process. But I think we are only scratching the surface of achieving this. Based on my admittedly casual observation of current model capabilities you need a model that's fairly big (closer to 70B than 7B) and trained far beyond what's implied by the Chinchilla metric.

caesil 2y ago

Why not exactly? The model weights cannot encode a sort of math engine? The hidden state cannot encode carryover values? Why do we assume these things can't happen at some level?

jasfi 2y ago
I agree. The reasoning is there, and becoming more capable every year (across the various models). It's easy to look for limitations, but what was once glaring problems are now much more subtle.