5 ms·

Mathematical reasoning is the most obvious area where it breaks down. This paper does an excellent job of proving this point with some elegant examples: https:/

Mathematical reasoning is the most obvious area where it breaks down. This paper does an excellent job of proving this point with some elegant examples: https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229

brookst 2y ago
Sure, but people fail at mathematical reasoning. That doesn't mean people are incapable of reasoning. I'm not saying LLMs are perfect reasoners, I'm questioning the value of asserting that they cannot reason with some kind of "it's just text that looks like reasoning" argument.
- dartos 2y ago
  People can communicate each step, and review each step as that communication is happening. LLMs must be prompted for everything and don’t act on their own. The value in the assertion is in preventing laymen from seeing a statistical guessing machine be correct and assuming that it always will be. It’s dangerous to put so much faith in what in reality is a very good guessing machine. You can ask it to retrace its steps, but it’s just guessing at what it’s steps were, since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps.
  - brookst 2y ago
    > since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps. Can you elaborate on the difference? Are you bringing sentience into it? It kind of sounds like it from "don't act on their own". But reasoning and sentience are wildly different things. > It’s dangerous to put so much faith in what in reality is a very good guessing machine Yes, exactly. That's why I think it is good we are supplementing fallible humans with fallible LLMs; we already have the processes in place to assume that not every actor is infallible.
    - david-gpu 2y ago
      So true. People who argue that we should not trust/use LLMs because they sometimes get it wrong are holding them to a higher standard than people -- we make mistakes too! Do we blindly trust or believe every single thing we hear from another person? Of course not. But hearing what they have to say can still be fruitful, and it is not like we have an oracle at our disposal who always speaks the absolute truth, either. We make do with what we have, and LLMs are another tool we can use.
    - vundercind 2y ago
      > Can you elaborate on the difference? They’ll fail in different ways than something that thinks (and doesn’t have some kind of major disease of the brain going on) and often smack in the middle of appearing to think.
  - ben_w 2y ago
    > People can communicate each step, and review each step as that communication is happening. Can, but don't by default. Just as LLMs can be asked for chain of thought, but the default for most users is just chat. This behaviour of humans is why we software developers have daily standup meetings, version control, and code review. > LLMs must be prompted for everything and don’t act on their own And this is why we humans have task boards like JIRA, and quarterly goals set by management.
  - int_19h 2y ago
    A human brain in a vat doesn't act on its own, either.
  - vidarh 2y ago
    LLMs "don't act on their own" because we only reanimate them when we want something from them. Nothing stops you from wiring up an LLM to keep generating, and feeding it sensory inputs to keep it processing. In other words, that's a limitation of the harness we put them in, not of LLMs. As for people communicating each step, we have plenty of experiments showing that it's pretty hard to get people to reliably report what they actually do as opposed to a rationalization of what they've actually done (e.g. split brain experiments have shown both your brain halves will happily lie about having decided to do things they haven't done if you give them reason to think they've done something) You can categorically not trust peoples reasoning about "why" they've made a decision to reflect what actually happened in their brain to make them do something.
- NBJack 2y ago
  The idea is the average person would, sure. A mathematically oriented person would fair far better. Throw all the math problems you want at a LLM for training; it will still fail if you step outside of the familiar.
  - ben_w 2y ago
    > it will still fail if you step outside of the familiar. To which I say: ᛋᛟ᛬ᛞᛟ᛬ᚻᚢᛗᚪᚾᛋ
    - trashtester 2y ago
      ᛒᚢᛏ ᚻᚢᛗᚪᚾ ᚻᚢᛒᚱᛁᛋ ᛈᚱᛖᚹᛖᚾᛏ ᚦᛖᛗ ᚠᚱᛟᛗ ᚱᛖᚪᛚᛁᛉᛁᚾᚷ ᚦᚻᚪᛏ
      - ben_w 2y ago
        ᛁᚾᛞᛖᛖᛞ᛬ᛁᛏ᛬ᛁᛋ᛬ᚻᚢᛒᚱᛁᛋ ᛁ᛬ᚻᚪᚹᛖ᛬ᛟᚠᛏᛖᚾ᛬ᛋᛖᛖᚾ᛬ᛁᚾ᛬ᛞᛁᛋᚲᚢᛋᛋᛁᛟᚾᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚦᛁᛋ᛬ᚲᛚᚪᛁᛗᛋ᛬ᚦᚪᛏ᛬ᚻᚢᛗᚪᚾ᛬ᛗᛁᚾᛞᛋ᛬ᚲᚪᚾ᛬ᛞᛟ᛬ᛁᛗᛈᛟᛋᛋᛁᛒᛚᛖ᛬ᚦᛁᛝᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚷᛖᚾᛖᚱᚪᛚᛚᚣ᛬ᛋᛟᛚᚹᛖ᛬ᚦᛖ᛬ᚻᚪᛚᛏᛁᛝ᛬ᛈᚱᛟᛒᛚᛖᛗ edit: Snap, you said the same in your other comment :)
        trashtester 2y ago
        Switching back to latin letters... It seems to me that the idea of the Universal Turing Machine is quite misleading for a lot of people, such as David Deutsch. My impression is that the amount of compute to solve most problems that can really only be solved by Turing Machines is always going to remain inaccessible (unless they're trivally small). But at the same time, the universe seems to obey a principle of locality (as long as we only consider the Quantum Wave Function, and don't postulate that it collapses). Also, the quantum fields are subject to some simple (relative to LLMs) geometric symmetries, such as invariance under the U(1)xSU(2)xSU(3) group. As it turns out, similar group symmetries can be found in all sorts of places in the real world. Also it seems to me that at some level, both ANN's and biological brains set up a similar system to this physical reality, which may explain why brains develop this way and why both kinds are so good at simulating at least some aspects of the physical world, such as translation, rotation, some types of deformation, gravity, sound, light etc. And when biological brains that initially developed to predict the physical world is then use to create language, that language is bound to use the same type of machinere. And this may be why LLM's do language so well with a similar architecture.
Workaccount2 2y ago
Maybe I am not understanding the paper correctly, but it seems they tested "state of the art models" which is almost entirely composed of open source <27B parameter models. Mostly 8B and 3B models. This is kind of like giving algebra problems to 7 year olds to "test human algebra ability." If you are holding up a 3B parameter model as an example of "LLM's can't reason" I'm not sure if the authors are confused or out of touch. I mean, they do test 4o and O1 preview, but their performance is notablely absent from the paper's conclusion.
- dartos 2y ago
  It’s difficult to reproducibly test openai models, since they can change from under you and you don’t have control over every hyperparameter. It would’ve been nice to see one of the larger llama models though.
  - famouswaffles 2y ago
    The results are there, it's just hidden away in the appendix. The result is that those models they don't actually suffer drops on 4/5 of their modified benchmarks. The one benchmark that does see actual drops that aren't explained by margin of error is the benchmark that adds "seemingly relevant but ultimately irrelevant information to problems" Those results are absent from the conclusion because the conclusion falls apart otherwise.

brookst 2y ago

Sure, but people fail at mathematical reasoning. That doesn't mean people are incapable of reasoning. I'm not saying LLMs are perfect reasoners, I'm questioning the value of asserting that they cannot reason with some kind of "it's just text that looks like reasoning" argument.

dartos 2y ago
People can communicate each step, and review each step as that communication is happening. LLMs must be prompted for everything and don’t act on their own. The value in the assertion is in preventing laymen from seeing a statistical guessing machine be correct and assuming that it always will be. It’s dangerous to put so much faith in what in reality is a very good guessing machine. You can ask it to retrace its steps, but it’s just guessing at what it’s steps were, since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps.
- brookst 2y ago
  > since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps. Can you elaborate on the difference? Are you bringing sentience into it? It kind of sounds like it from "don't act on their own". But reasoning and sentience are wildly different things. > It’s dangerous to put so much faith in what in reality is a very good guessing machine Yes, exactly. That's why I think it is good we are supplementing fallible humans with fallible LLMs; we already have the processes in place to assume that not every actor is infallible.
  - david-gpu 2y ago
    So true. People who argue that we should not trust/use LLMs because they sometimes get it wrong are holding them to a higher standard than people -- we make mistakes too! Do we blindly trust or believe every single thing we hear from another person? Of course not. But hearing what they have to say can still be fruitful, and it is not like we have an oracle at our disposal who always speaks the absolute truth, either. We make do with what we have, and LLMs are another tool we can use.
  - vundercind 2y ago
    > Can you elaborate on the difference? They’ll fail in different ways than something that thinks (and doesn’t have some kind of major disease of the brain going on) and often smack in the middle of appearing to think.
- ben_w 2y ago
  > People can communicate each step, and review each step as that communication is happening. Can, but don't by default. Just as LLMs can be asked for chain of thought, but the default for most users is just chat. This behaviour of humans is why we software developers have daily standup meetings, version control, and code review. > LLMs must be prompted for everything and don’t act on their own And this is why we humans have task boards like JIRA, and quarterly goals set by management.
- int_19h 2y ago
  A human brain in a vat doesn't act on its own, either.
- vidarh 2y ago
  LLMs "don't act on their own" because we only reanimate them when we want something from them. Nothing stops you from wiring up an LLM to keep generating, and feeding it sensory inputs to keep it processing. In other words, that's a limitation of the harness we put them in, not of LLMs. As for people communicating each step, we have plenty of experiments showing that it's pretty hard to get people to reliably report what they actually do as opposed to a rationalization of what they've actually done (e.g. split brain experiments have shown both your brain halves will happily lie about having decided to do things they haven't done if you give them reason to think they've done something) You can categorically not trust peoples reasoning about "why" they've made a decision to reflect what actually happened in their brain to make them do something.
NBJack 2y ago
The idea is the average person would, sure. A mathematically oriented person would fair far better. Throw all the math problems you want at a LLM for training; it will still fail if you step outside of the familiar.
- ben_w 2y ago
  > it will still fail if you step outside of the familiar. To which I say: ᛋᛟ᛬ᛞᛟ᛬ᚻᚢᛗᚪᚾᛋ
  - trashtester 2y ago
    ᛒᚢᛏ ᚻᚢᛗᚪᚾ ᚻᚢᛒᚱᛁᛋ ᛈᚱᛖᚹᛖᚾᛏ ᚦᛖᛗ ᚠᚱᛟᛗ ᚱᛖᚪᛚᛁᛉᛁᚾᚷ ᚦᚻᚪᛏ
    - ben_w 2y ago
      ᛁᚾᛞᛖᛖᛞ᛬ᛁᛏ᛬ᛁᛋ᛬ᚻᚢᛒᚱᛁᛋ ᛁ᛬ᚻᚪᚹᛖ᛬ᛟᚠᛏᛖᚾ᛬ᛋᛖᛖᚾ᛬ᛁᚾ᛬ᛞᛁᛋᚲᚢᛋᛋᛁᛟᚾᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚦᛁᛋ᛬ᚲᛚᚪᛁᛗᛋ᛬ᚦᚪᛏ᛬ᚻᚢᛗᚪᚾ᛬ᛗᛁᚾᛞᛋ᛬ᚲᚪᚾ᛬ᛞᛟ᛬ᛁᛗᛈᛟᛋᛋᛁᛒᛚᛖ᛬ᚦᛁᛝᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚷᛖᚾᛖᚱᚪᛚᛚᚣ᛬ᛋᛟᛚᚹᛖ᛬ᚦᛖ᛬ᚻᚪᛚᛏᛁᛝ᛬ᛈᚱᛟᛒᛚᛖᛗ edit: Snap, you said the same in your other comment :)
      - trashtester 2y ago
        Switching back to latin letters... It seems to me that the idea of the Universal Turing Machine is quite misleading for a lot of people, such as David Deutsch. My impression is that the amount of compute to solve most problems that can really only be solved by Turing Machines is always going to remain inaccessible (unless they're trivally small). But at the same time, the universe seems to obey a principle of locality (as long as we only consider the Quantum Wave Function, and don't postulate that it collapses). Also, the quantum fields are subject to some simple (relative to LLMs) geometric symmetries, such as invariance under the U(1)xSU(2)xSU(3) group. As it turns out, similar group symmetries can be found in all sorts of places in the real world. Also it seems to me that at some level, both ANN's and biological brains set up a similar system to this physical reality, which may explain why brains develop this way and why both kinds are so good at simulating at least some aspects of the physical world, such as translation, rotation, some types of deformation, gravity, sound, light etc. And when biological brains that initially developed to predict the physical world is then use to create language, that language is bound to use the same type of machinere. And this may be why LLM's do language so well with a similar architecture.

Workaccount2 2y ago

Maybe I am not understanding the paper correctly, but it seems they tested "state of the art models" which is almost entirely composed of open source <27B parameter models. Mostly 8B and 3B models. This is kind of like giving algebra problems to 7 year olds to "test human algebra ability." If you are holding up a 3B parameter model as an example of "LLM's can't reason" I'm not sure if the authors are confused or out of touch. I mean, they do test 4o and O1 preview, but their performance is notablely absent from the paper's conclusion.

dartos 2y ago
It’s difficult to reproducibly test openai models, since they can change from under you and you don’t have control over every hyperparameter. It would’ve been nice to see one of the larger llama models though.
- famouswaffles 2y ago
  The results are there, it's just hidden away in the appendix. The result is that those models they don't actually suffer drops on 4/5 of their modified benchmarks. The one benchmark that does see actual drops that aren't explained by margin of error is the benchmark that adds "seemingly relevant but ultimately irrelevant information to problems" Those results are absent from the conclusion because the conclusion falls apart otherwise.