4 ms·

Reasoning skills of large language models are often overestimated

37 points by meysamazad 2y ago

rco8786 2y ago
Because there is no "reasoning" right? LLMs create output that appear to have been reasoned about, but this is not at all reality
- nathan_compton 2y ago
  I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.
  - onemoresoop 2y ago
    That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going
    - bckr 2y ago
      Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
      - swatcoder 2y ago
        Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
        onemoresoop 2y ago
        > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
        Ekaros 2y ago
        I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
    - JustBreath 2y ago
      Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.
- cmrdporcupine 2y ago
  I agree, but apologists will say that that's all the human brain is doing, too. Myself, I think that's a terrible false analogy, and also have a higher bar for anything to be called artificial "intelligence." These programs shouldn't be using personal pronouns or statements that imply cognition or esp reasoning. It's deceptive.
  - AnimalMuppet 2y ago
    I think nathan_compton is right in describing LLMs as "a single pass, feed forward, process calculating outputs from inputs". That is not all the human brain is doing. Human reasoning is not single pass - far from it. So I concur that the apologists are wrong about what the human brain is doing, and therefore wrong about the similarity to LLMs.
    - Kim_Bruning 2y ago
      If an LLM gets extra passes, it can be smarter. This is especially important for eg. strict logic puzzles, where error correction is needed.
    - cmrdporcupine 2y ago
      My gut instinct / guess is that there's "stuff" in the brain that is a kind of system for doing the kind of thing that symbolic logic systems do: proposition management, unification, backtracking, etc. And I think this stuff is probably there from a very early age. Also, generally, while humans can be really bad at reasoning, when they're shown why they're wrong they're able to retrace their steps and fix -- assuming they're not emotionally attached to some outcome. I'm terrible at math, but if I slow down and have it explained, I'll get it (and then forget it.) LLMs... they don't work that way. Tell them how they're wrong or to explain how they got to some result... forget it, they can't retrace and fix.
      - Kim_Bruning 2y ago
        I posted several conversations with models in this thread. I did -in fact- manage to get them to retrace and fix where needed. GPT-4o was a lot of work. GPT-3 is fast but that was a long conversation catching all its errors (so not posted, but I can if you insist). 4 caught its error almost immediately. Claude-3.5 was on to us from the get-go and didn't need correcting.
- wongarsu 2y ago
  In principle you would expect reasoning to eventually emerge. During training LLMs predict large volumes of text, and that text contains a lot of arguments and chains of thought. You can achieve some success predicting these with pattern matching, but the best way to complete arguments at scale is a reasoning process. But I think we are only scratching the surface of achieving this. Based on my admittedly casual observation of current model capabilities you need a model that's fairly big (closer to 70B than 7B) and trained far beyond what's implied by the Chinchilla metric.
- caesil 2y ago
  Why not exactly? The model weights cannot encode a sort of math engine? The hidden state cannot encode carryover values? Why do we assume these things can't happen at some level?
  - jasfi 2y ago
    I agree. The reasoning is there, and becoming more capable every year (across the various models). It's easy to look for limitations, but what was once glaring problems are now much more subtle.
shermantanktop 2y ago
Chord fingering is mentioned. I’ve asked ChatGPT questions about drop-2 voicings and it explained drop-2 and then gave examples of drop-3. I told it that it was in correct and it then changed some wording but repeated the incorrect examples.
- bckr 2y ago
  ChatGPT is terrible with music-related prompts. It might be able to get some basic theory, the type you can read about in Wikipedia. But ask it to do anything beyond that and it will fail pretty hard.
apsec112 2y ago
FWIW the version of Claude they use is quite old, v1.3. I'd be curious to see how the new Claude 3.5 performs on these
ArcaneMoose 2y ago
This is why I find ARC-AGI so fascinating - it really seems like a great benchmark to test for this sort of behavior. Hope that it doesn't get solved through a brute-force memorization approach. I do wonder how reasoning is different from other sorts of 'thinking' if it's all neurons firing in the brain.
alexsereno 2y ago
I’d love to see this placed into the context of the (actual) average Joe. If you consider that the average person dramatically overestimates their reasoning skills and can’t correctly do high school level math, I think it would help understand if we have reached AGI. It would also just provide some color to the state of things.
deleted 2y ago
[deleted]
armitron 2y ago
The "GPT-4 Can't Reason" paper [1] [2] is excellent and proves without a shadow of a doubt that there is no reasoning taking place. It also raises strong doubts about the efficacy and reliability of multi-agent approaches that rely on LLM-based planning (such as Cradle [3], which is currently on the front page). When it was previously discussed here, it received a torrent of low-quality comments that can only be described as confirmation bias: commenters tried the examples verbatim (instead of introducing random perturbations to get around model updates) in order to disprove the thesis. It's sad that there are "engineers" out there so blinded by their own (wishful thinking, vested interests) that can not accept the obvious. [1] https://arxiv.org/abs/2308.03762 https://arxiv.org/abs/2308.03762 [2] https://medium.com/@konstantine_45825/gpt-4-cant-reason-addendum-ed79d8452d44 https://medium.com/@konstantine_45825/gpt-4-cant-reason-adde... [3] https://baai-agents.github.io/Cradle/ https://baai-agents.github.io/Cradle/
- krageon 2y ago
  I think most people are a little bit tired of hearing that things that are manifestly possible, are instead impossible. If we'd take the time to recognise that the truth is somewhere in the middle (lots of tasks can be done, lots of them cannot), the conversation could at the very least be interesting. As it stands every publication on this subject is filled with a lot of people that feel they need a polarised stance or they'll immediately be shot: Either they say LLMs are fundamentally broken and can't do shit (this is ridiculous) or they say that LLMs are self-aware and we should give them every single task ever (also ridiculous). It is really tedious to read the same dumb shit every single time.
  - mewpmewp2 2y ago
    If any LLM can solve a novel exercise that requires some form of reasoning, wouldn't that be evidence that LLMs are capable of at least some level of reasoning? Not necessarily AGI or average human level reasoning, but just any form.
- deleted 2y ago
  [deleted]
cs702 2y ago
Paper: https://arxiv.org/abs/2407.02678 https://arxiv.org/abs/2407.02678 Previous discussion: https://news.ycombinator.com/item?id=40899309 https://news.ycombinator.com/item?id=40899309
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

37 points by meysamazad 2y ago

Because there is no "reasoning" right? LLMs create output that appear to have been reasoned about, but this is not at all reality

nathan_compton 2y ago
I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.
- onemoresoop 2y ago
  That's right I did this on chat gpt and went off the rails quickly: a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river? The response was: To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps: 1.Man takes the sheep across the river (leaving the wolf and the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the sheep back 5.Man takes the wolf across the river. 6.Man returns alone. 7.Man takes the sheep across the river. 8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together. at which I prompted: I didn't mention a sheep The response was corrected but wrongly: Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage: 1.Man takes the wolf across the river (leaving the cabbage on the original side). 2.Man returns alone. 3.Man takes the cabbage across the river. 4.Man leaves the cabbage on the other side and takes the wolf back. 5.Man takes the man across the river. 6.Man returns alone. 7.Man takes the wolf across the river. 8.Now, both the man and the wolf are across the river, with no danger to the cabbage. At step 3 the problem was solved but it kept on going
  - bckr 2y ago
    Oh wow I got similar results with GPT4o. I’m surprised! It even messed up the second round in a similar way.
    - swatcoder 2y ago
      Presumably, because river crossing challenges are an important benchmark in satisfying AI hype, it's been aggressively trained to output solutions to them and now overfits their three-object/eight-crossing structure onto prompts that approximate the river cross prompt. Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes.
      - onemoresoop 2y ago
        > Parlor tricks are lurking everywhere, and especially in ChatGPT, because of it's high profile and high stakes. The only way for LLMs to get this right is to be trained on all possible paths of all possible problems, which is not very feasible. That's not really reasoning, you may as well calculate all permutations and add it to a DB and call it a day.
      - Ekaros 2y ago
        I think it is more so just very common data in any dataset used in training. And thus you would expect model to produce it when relevant tokens appear. These models store these common patterns and then repeat slight variations of them as that is natural flow of the tokens. Humans do it similarly with the same problem if they have heard it before. Challenge is that coming up with actually novel reasoning problems is not that easy. So generating one to test models is non-trivial task.
  - JustBreath 2y ago
    Yeah, exactly, it isn't reasoning or creating in the way that humans reason, it's calculating that statistically speaking, when a human says what you said here's roughly what often comes next. It's like debating your political other. You can often predict what they will say not because you understand their reasoning, but simply because you've seen multiple times how they react to a given argument in the past.
cmrdporcupine 2y ago
I agree, but apologists will say that that's all the human brain is doing, too. Myself, I think that's a terrible false analogy, and also have a higher bar for anything to be called artificial "intelligence." These programs shouldn't be using personal pronouns or statements that imply cognition or esp reasoning. It's deceptive.
- AnimalMuppet 2y ago
  I think nathan_compton is right in describing LLMs as "a single pass, feed forward, process calculating outputs from inputs". That is not all the human brain is doing. Human reasoning is not single pass - far from it. So I concur that the apologists are wrong about what the human brain is doing, and therefore wrong about the similarity to LLMs.
  - Kim_Bruning 2y ago
    If an LLM gets extra passes, it can be smarter. This is especially important for eg. strict logic puzzles, where error correction is needed.
  - cmrdporcupine 2y ago
    My gut instinct / guess is that there's "stuff" in the brain that is a kind of system for doing the kind of thing that symbolic logic systems do: proposition management, unification, backtracking, etc. And I think this stuff is probably there from a very early age. Also, generally, while humans can be really bad at reasoning, when they're shown why they're wrong they're able to retrace their steps and fix -- assuming they're not emotionally attached to some outcome. I'm terrible at math, but if I slow down and have it explained, I'll get it (and then forget it.) LLMs... they don't work that way. Tell them how they're wrong or to explain how they got to some result... forget it, they can't retrace and fix.
    - Kim_Bruning 2y ago
      I posted several conversations with models in this thread. I did -in fact- manage to get them to retrace and fix where needed. GPT-4o was a lot of work. GPT-3 is fast but that was a long conversation catching all its errors (so not posted, but I can if you insist). 4 caught its error almost immediately. Claude-3.5 was on to us from the get-go and didn't need correcting.
wongarsu 2y ago
In principle you would expect reasoning to eventually emerge. During training LLMs predict large volumes of text, and that text contains a lot of arguments and chains of thought. You can achieve some success predicting these with pattern matching, but the best way to complete arguments at scale is a reasoning process. But I think we are only scratching the surface of achieving this. Based on my admittedly casual observation of current model capabilities you need a model that's fairly big (closer to 70B than 7B) and trained far beyond what's implied by the Chinchilla metric.
caesil 2y ago
Why not exactly? The model weights cannot encode a sort of math engine? The hidden state cannot encode carryover values? Why do we assume these things can't happen at some level?
- jasfi 2y ago
  I agree. The reasoning is there, and becoming more capable every year (across the various models). It's easy to look for limitations, but what was once glaring problems are now much more subtle.

shermantanktop 2y ago

Chord fingering is mentioned. I’ve asked ChatGPT questions about drop-2 voicings and it explained drop-2 and then gave examples of drop-3. I told it that it was in correct and it then changed some wording but repeated the incorrect examples.

bckr 2y ago
ChatGPT is terrible with music-related prompts. It might be able to get some basic theory, the type you can read about in Wikipedia. But ask it to do anything beyond that and it will fail pretty hard.

apsec112 2y ago

FWIW the version of Claude they use is quite old, v1.3. I'd be curious to see how the new Claude 3.5 performs on these

ArcaneMoose 2y ago

This is why I find ARC-AGI so fascinating - it really seems like a great benchmark to test for this sort of behavior. Hope that it doesn't get solved through a brute-force memorization approach. I do wonder how reasoning is different from other sorts of 'thinking' if it's all neurons firing in the brain.

alexsereno 2y ago

I’d love to see this placed into the context of the (actual) average Joe. If you consider that the average person dramatically overestimates their reasoning skills and can’t correctly do high school level math, I think it would help understand if we have reached AGI. It would also just provide some color to the state of things.

deleted 2y ago

[deleted]

armitron 2y ago

The "GPT-4 Can't Reason" paper [1] [2] is excellent and proves without a shadow of a doubt that there is no reasoning taking place. It also raises strong doubts about the efficacy and reliability of multi-agent approaches that rely on LLM-based planning (such as Cradle [3], which is currently on the front page). When it was previously discussed here, it received a torrent of low-quality comments that can only be described as confirmation bias: commenters tried the examples verbatim (instead of introducing random perturbations to get around model updates) in order to disprove the thesis. It's sad that there are "engineers" out there so blinded by their own (wishful thinking, vested interests) that can not accept the obvious. [1] https://arxiv.org/abs/2308.03762 https://arxiv.org/abs/2308.03762 [2] https://medium.com/@konstantine_45825/gpt-4-cant-reason-addendum-ed79d8452d44 https://medium.com/@konstantine_45825/gpt-4-cant-reason-adde... [3] https://baai-agents.github.io/Cradle/ https://baai-agents.github.io/Cradle/

krageon 2y ago
I think most people are a little bit tired of hearing that things that are manifestly possible, are instead impossible. If we'd take the time to recognise that the truth is somewhere in the middle (lots of tasks can be done, lots of them cannot), the conversation could at the very least be interesting. As it stands every publication on this subject is filled with a lot of people that feel they need a polarised stance or they'll immediately be shot: Either they say LLMs are fundamentally broken and can't do shit (this is ridiculous) or they say that LLMs are self-aware and we should give them every single task ever (also ridiculous). It is really tedious to read the same dumb shit every single time.
- mewpmewp2 2y ago
  If any LLM can solve a novel exercise that requires some form of reasoning, wouldn't that be evidence that LLMs are capable of at least some level of reasoning? Not necessarily AGI or average human level reasoning, but just any form.
deleted 2y ago
[deleted]

cs702 2y ago

Paper: https://arxiv.org/abs/2407.02678 https://arxiv.org/abs/2407.02678 Previous discussion: https://news.ycombinator.com/item?id=40899309 https://news.ycombinator.com/item?id=40899309

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]