7 ms·

Why are transformer models so bad at math? They often fail at simple addition.

jjtheblunt 3y ago
Don't they essentially memorize all examples they've seen, which contrasts with understanding how to solve problems as in addition with carries on digits etc?
- tsunamifury 3y ago
  No, stop repeating this. They use examples to learn and attempt to generalize with on-the-fly modeling.
  - jjtheblunt 3y ago
    Are you claiming the on the fly modeling can learn or reproduce multistep algorithms, like simple addition that the parent comment asked about, and which i asked about above?
    - vidarh 3y ago
      I just fed it this: --- Here is a grammar for a made up language: sentence = "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). verb = "ab" | "pom" object = "foo" | "bar" Construct a few sentences, and show the sentence so far, and which rule and step of the rule you are currently applying. E.g. sentence= ""; rule=sentence; step="sob". --- Notice I did not explain the BNF syntax, which is reasonable as it'll have plenty of that in its training data. Notice that I also "forgot" to give it a production for the "subject" rule - leaving it with two choices: It can give up and tell me off, or it can do as best it can and let me know it's had to improvise. Here's the first sentence construction: --- First Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: (choosing object as a subject for simplicity) "foo". sentence = "sob foo"; Rule: verb; Step: "ab". sentence = "sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob foo ab ob"; Rule: object; Step: "bar". sentence = "sob foo ab ob bar"; (Complete Sentence) --- It has correctly applied the rules I did give, and opted to "fill in" "object" as a replacement for my missing "subject" rule. There are few enough rules this could be a fluke, but it's a better choice to guess at that substitution than verb or sentence, though hardly takes a genius. Second: --- Second Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "bar". sentence = "sob bar"; Rule: verb; Step: "pom". sentence = "sob bar pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ad". sentence = "sob bar pom ad"; Rule: sentence; Step: "sob" (this starts a nested sentence). sentence = "sob bar pom ad sob"; Rule: subject; Step: "foo". sentence = "sob bar pom ad sob foo"; Rule: verb; Step: "ab". sentence = "sob bar pom ad sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob bar pom ad sob foo ab ob"; Rule: object; Step: "foo". sentence = "sob bar pom ad sob foo ab ob foo"; (Complete Sentence) --- This is also correct, and it correctly points out that the arm it picked triggered a nested application of the `sentence` rule. It stumbled on the third attempt: --- Third Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "foo". sentence = "sob foo"; Rule: verb; Step: "pom". sentence = "sob foo pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ploj". sentence = "sob foo pom ploj"; (Complete Sentence) --- Note that there is a possible interpretation of the "|" operator which would make its sentence valid, so "technically" it could be, but it would involve (ver "ob" object | verb "ad" sentence | "ploj") to be interpreted as (verb (("ob object)|("ad" sentence)|"ploj")) and that's contrived. So to answer you: It can, and it does, but not always consistently. One can choose to be disappointed it's not perfect, but for just handing it a grammar and asking it to reason about it with no example text or chances to "practice" whatsoever, I think it's doing quite well. I've interviewed many software developers who'd struggle with a task like this. When prodded about the third example, it picks a more reasonable interpretation of the "|", though it's explanation starts out with an error, but then given a correct explanation: --- Q: How do you figure this? The full rule is "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). How do you figure the verb is allowed without being followed by either "ad" or "ob"? A: Upon closer examination, you are correct to point out that the structure following "sob" subject must indeed involve a verb according to the rule provided. The full rule: "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). indicates that the sentence must start with "sob", followed by a subject, and then either: a verb followed by "ob" and an object, a verb followed by "ad" and another sentence, or the standalone term "ploj". --- It goes on to give two correct alternatives to the erroneous one: --- * "sob foo ploj" — where "ploj" follows directly after the subject without a verb, or * "sob foo ab ob bar" — where "ab" is the verb followed by "ob" and "bar" as the object. --- While it's not perfect, it can very clearly "learn and reproduce multistep algorithms" to some extent. It also clearly needs to get better at it.
      - tsunamifury 3y ago
        Yea I mean you don’t even have to go this far but it’s obvious it can do logical, mathematical and systematic reasoning. I have no idea why people keep insisting it can’t.
        vidarh 3y ago
        Yeah, I asked it that partly because I got curious if it could handle a task that well (because I know plenty of people who can't...), and partly because so many people keep rejecting other reasoning tasks.
        jjtheblunt 3y ago
        People keep asking because they are looking for a rigorous example in a citation that's unambiguous. If it's not obvious, that is even more compelling. I personally think the more theory trained folks wonder if there will turn out to be some analogy with the corresponding towers of languages and machines recognizing them (regular languages, finite automata, push down automata, turing machines, for examples). https://a.co/d/8ix6V6m https://a.co/d/8ix6V6m is an example covering the historical language classes and recognizers for them. Perhaps LLMs and various architectures will turn out fitting what you describe.
  - idontpost 3y ago
    [dead]
  - jjtheblunt 3y ago
    Citation instead of handwaving that they can learn algorithms? Also, I’ve not asked that before so if it’s being asked frequently, I would imagine it might be because it’s a good question.
- GaggiX 3y ago
  No they don't.
  - ctoth 3y ago
    Yes, actually, they do. See the heading Blessings of scale in [0] [0]: https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis:
    - GaggiX 3y ago
      I have trained models on performing additions on numbers with fixed number of digits, just to prove to a friend that a neural network can learn to do addition, I hide in the dataset many combinations of numbers but the model was still able to sum them correctly, therefore it learned to perform addition on numbers with fixed number of digits. So no it's not memorizing, it's something that I have tested myself.
      - jjtheblunt 3y ago
        Interesting. Thanks for the comment. What if you ask it to add numbers with more digits?
        GaggiX 3y ago
        The model is trained with a fixed number of tokens, I don't remember if the models I trained have sinusoidal embeddings or learnable positional embeddings, in the latter there would be no embedding to encode the position, in the former I think it would cause problems with the sinusoidal embedding layer as the sine and cosine would wrap around.
      - Der_Einzige 3y ago
        Fine-tuning the model to force it to understand the insane tokenization its forced to use for its vocabulary and perform accurate addition sounds great until you start trying very large numbers or messing with whatever decoding settings you're using (and let me guess, it was beam search). Or, you can simply ask it to use a tool, like a calculator for you. This is more reliable than fine-tuning a current technique BPE tokenized model ever will be.
- zamadatix 3y ago
  This oversimplifies too much. GPT4 is able to add random large numbers too large to have been in the data set. When it's wrong it tends to be closely wrong e.g. maybe the first 4 digits of division on a large number by a decimal number with many fractional digits are correct. Neither of these examples can be explained by memorization alone, other operational mechanics are being applied to get these outcomes.
  - jjtheblunt 3y ago
    To your point, it would be really neat if we could trace what it's doing to get answers, see why it's sometimes right, sometimes close but wrong!
    - zamadatix 3y ago
      You can trace it pretty easily in a local neural net which takes a couple of minutes to train. The only reason we can't trace it in GPT is it's a closed model so we don't have access to do so. Largely two things come into play: 1) Some part of the neural net is emulating more traditional logic, but it may not always be the most activated part or tuned to be perfect in the answers 2) There isn't really a "jmp" equivalent in a single iteration, so the neural net has to learn to not only do decimal division but do it based on iterating output tokens, continuing perfectly each token output, and choosing to put the right stuff into context and keep that context activated at the right time. "Activated" in this case means, more or less, the group of neurons specializing on this task are being both fed and listened to. You can even train a neural net to emulate a traditional addition circuit directly, it's just less efficient than one would think if you're trying to build a general purpose model instead of a specialized one.
haltist 3y ago
Neural networks approximate the statistical distribution of whatever they are trained on. There is no reason to expect that any neural network will perform well on out of distribution tasks, e.g. multi-digit addition and multiplication.
- ctoth 3y ago
  This is (probably) wrong, see: https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis And specifically the part where it discusses addition under the heading Blessings of Scale.
  - haltist 3y ago
    So why can't GPT do arithmetic? At what scale do you suppose it will manage to add numbers?
    - famouswaffles 3y ago
      GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.
      - Der_Einzige 3y ago
        Besides your answer on tokenization being right on the mark (gwern even has articles discussing this exact issue - https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes), that linked paper is extremely powerful for theoretical thinking about this problem. I literally just left a comment claiming that almost all of human thought is encoded in LLMs, leading almost all tasks to be considered interpolation. If this is true, I suppose based on what we know of the curse of dimensionality it makes sense. Very thought-provoking work.
        famouswaffles 3y ago
tsunamifury 3y ago
They aren't. This is overblown if you ask them to do simple math alone they are fine. And what they can't do they can do better with reflexion. It's not perfect, but they aren't terrible.
- simbolit 3y ago
  Just one (of many) examples: I asked for scores in several different categories, and the total score was not what the individiual scores added up to. I mentioned this, the model apologized, and gave another total score (also wrong).
GaggiX 3y ago
If you give them a calculator they will be much better. A better answer: every time the model needs to predict the digit of a sum the model needs to solve the entire addition to know the carries, a bigger sum requires a bigger model to solve them.
ctoth 3y ago
Tokenization, same as a year ago.
craftit 3y ago
Because often, the tokens are broken up as random groups of numbers. For example, let's say 1984 appears quite a few times in the source text, this will become a single token. Given that these many different, semi-random groups of digits it is hard for the LLM to learn any consistent rules. I believe there are papers showing that if you structure numbers more consistently LLMs have no problem with this kind of arithmetic.
- ffwd 3y ago
  I have a question which I don't know the answer to: With those structured numbers will the LLMs be 100% accurate on new prompts or will they just be better than chance (even significantly better than chance)? Because this is one thing, it has to learn the structure and then create probabilities based on the data, but does that mean it's actually learning the underlying algorithm for addition for example or is it just getting better probabilities because of a narrowing of them? If it can indeed learn underlying algorithms like this that's super interesting. The reason also this is in an issue if it _can't_ learn those, you can never trust the answer unless you check it, but that's sort of a sidepoint.
  - craftit 3y ago
    From what I understand, it can learn and execute the algorithm fairly reliably, though it won't be 100%. When the LLM generates text, it is randomised a little, as well as some tricks that prevent repetition, which would likely cause problems with numbers containing all the same digit.
vidarh 3y ago
Why are human children so bad at math until they've repetitively trained on how to apply each basic operation over and over and over even after having learned rules that should be simple to apply? As it happens, conditioning even people to actually apply rules properly tends to take a lot of repetition. How many individual examples of step-by-step working out basic math problems do you think they have been in their training data? Prompt them the way you'd prompt a child who is sloppy into working step by step and explaining how it applies the rules it has learned, and it will tend to do better. While tokenization might not help, I don't think there's an inherent problem there beyond feeding them enough training data. Whether that's worthwhile vs. having it resort to tools, is another matter.
Verdex 3y ago
I'm definitely not on the LLM bandwagon. While I do think they're interesting, I have significant doubts that they'll be much more than a curiosity to laymen and just one more tool in the box for experts. However, I'm not sure why being bad at math (if even true based on other comments here), is a legitimate criticism. We've already got a lot of machinery that is very good at math. So, use the tools that make sense for the domain. Or (what I'm really waiting for) better yet find novel ways to stitch the tools that we do have together. Can LLMs turn a problem statement into a matlab program? If so then it doesn't really matter how bad at math they are.
spywaregorilla 3y ago
Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible.
- famouswaffles 3y ago
  This is wrong. The token predictor will learn algorithms for math. You can ask GPT-4 arbitrary arithmetic it could never see in it's training set. Even when it's not completely correct, it's extremely close. It is clearly computing algorithms even if those algorithms are not quite right.
  - spywaregorilla 3y ago
    It is clearly not computing these algorithms. Getting close is more evidence of that than otherwise. Being able to infer answers to math problems is a thing that humans can do, and that's fine. It's not as good as doing the math though.
    - famouswaffles 3y ago
      It is computing algorithms. There's no debate here. The question is whether it is computing the right ones.
      - spywaregorilla 3y ago
        It is not computing the right ones. There is no debate there. These algorithms do not map inputs to outputs reliably.
        famouswaffles 3y ago
        The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool or pad. The biggest problem with LLM arithmetic is tokenization. https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Other than that, the algorithms it uses for arithmetic will continue to converge during training until it is correct.

Don't they essentially memorize all examples they've seen, which contrasts with understanding how to solve problems as in addition with carries on digits etc?

tsunamifury 3y ago
No, stop repeating this. They use examples to learn and attempt to generalize with on-the-fly modeling.
- jjtheblunt 3y ago
  Are you claiming the on the fly modeling can learn or reproduce multistep algorithms, like simple addition that the parent comment asked about, and which i asked about above?
  - vidarh 3y ago
    I just fed it this: --- Here is a grammar for a made up language: sentence = "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). verb = "ab" | "pom" object = "foo" | "bar" Construct a few sentences, and show the sentence so far, and which rule and step of the rule you are currently applying. E.g. sentence= ""; rule=sentence; step="sob". --- Notice I did not explain the BNF syntax, which is reasonable as it'll have plenty of that in its training data. Notice that I also "forgot" to give it a production for the "subject" rule - leaving it with two choices: It can give up and tell me off, or it can do as best it can and let me know it's had to improvise. Here's the first sentence construction: --- First Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: (choosing object as a subject for simplicity) "foo". sentence = "sob foo"; Rule: verb; Step: "ab". sentence = "sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob foo ab ob"; Rule: object; Step: "bar". sentence = "sob foo ab ob bar"; (Complete Sentence) --- It has correctly applied the rules I did give, and opted to "fill in" "object" as a replacement for my missing "subject" rule. There are few enough rules this could be a fluke, but it's a better choice to guess at that substitution than verb or sentence, though hardly takes a genius. Second: --- Second Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "bar". sentence = "sob bar"; Rule: verb; Step: "pom". sentence = "sob bar pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ad". sentence = "sob bar pom ad"; Rule: sentence; Step: "sob" (this starts a nested sentence). sentence = "sob bar pom ad sob"; Rule: subject; Step: "foo". sentence = "sob bar pom ad sob foo"; Rule: verb; Step: "ab". sentence = "sob bar pom ad sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob bar pom ad sob foo ab ob"; Rule: object; Step: "foo". sentence = "sob bar pom ad sob foo ab ob foo"; (Complete Sentence) --- This is also correct, and it correctly points out that the arm it picked triggered a nested application of the `sentence` rule. It stumbled on the third attempt: --- Third Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "foo". sentence = "sob foo"; Rule: verb; Step: "pom". sentence = "sob foo pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ploj". sentence = "sob foo pom ploj"; (Complete Sentence) --- Note that there is a possible interpretation of the "|" operator which would make its sentence valid, so "technically" it could be, but it would involve (ver "ob" object | verb "ad" sentence | "ploj") to be interpreted as (verb (("ob object)|("ad" sentence)|"ploj")) and that's contrived. So to answer you: It can, and it does, but not always consistently. One can choose to be disappointed it's not perfect, but for just handing it a grammar and asking it to reason about it with no example text or chances to "practice" whatsoever, I think it's doing quite well. I've interviewed many software developers who'd struggle with a task like this. When prodded about the third example, it picks a more reasonable interpretation of the "|", though it's explanation starts out with an error, but then given a correct explanation: --- Q: How do you figure this? The full rule is "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). How do you figure the verb is allowed without being followed by either "ad" or "ob"? A: Upon closer examination, you are correct to point out that the structure following "sob" subject must indeed involve a verb according to the rule provided. The full rule: "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). indicates that the sentence must start with "sob", followed by a subject, and then either: a verb followed by "ob" and an object, a verb followed by "ad" and another sentence, or the standalone term "ploj". --- It goes on to give two correct alternatives to the erroneous one: --- * "sob foo ploj" — where "ploj" follows directly after the subject without a verb, or * "sob foo ab ob bar" — where "ab" is the verb followed by "ob" and "bar" as the object. --- While it's not perfect, it can very clearly "learn and reproduce multistep algorithms" to some extent. It also clearly needs to get better at it.
    - tsunamifury 3y ago
      Yea I mean you don’t even have to go this far but it’s obvious it can do logical, mathematical and systematic reasoning. I have no idea why people keep insisting it can’t.
      - vidarh 3y ago
        Yeah, I asked it that partly because I got curious if it could handle a task that well (because I know plenty of people who can't...), and partly because so many people keep rejecting other reasoning tasks.
      - jjtheblunt 3y ago
        People keep asking because they are looking for a rigorous example in a citation that's unambiguous. If it's not obvious, that is even more compelling. I personally think the more theory trained folks wonder if there will turn out to be some analogy with the corresponding towers of languages and machines recognizing them (regular languages, finite automata, push down automata, turing machines, for examples). https://a.co/d/8ix6V6m https://a.co/d/8ix6V6m is an example covering the historical language classes and recognizers for them. Perhaps LLMs and various architectures will turn out fitting what you describe.
- idontpost 3y ago
  [dead]
- jjtheblunt 3y ago
  Citation instead of handwaving that they can learn algorithms? Also, I’ve not asked that before so if it’s being asked frequently, I would imagine it might be because it’s a good question.
GaggiX 3y ago
No they don't.
- ctoth 3y ago
  Yes, actually, they do. See the heading Blessings of scale in [0] [0]: https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis:
  - GaggiX 3y ago
    I have trained models on performing additions on numbers with fixed number of digits, just to prove to a friend that a neural network can learn to do addition, I hide in the dataset many combinations of numbers but the model was still able to sum them correctly, therefore it learned to perform addition on numbers with fixed number of digits. So no it's not memorizing, it's something that I have tested myself.
    - jjtheblunt 3y ago
      Interesting. Thanks for the comment. What if you ask it to add numbers with more digits?
      - GaggiX 3y ago
        The model is trained with a fixed number of tokens, I don't remember if the models I trained have sinusoidal embeddings or learnable positional embeddings, in the latter there would be no embedding to encode the position, in the former I think it would cause problems with the sinusoidal embedding layer as the sine and cosine would wrap around.
    - Der_Einzige 3y ago
      Fine-tuning the model to force it to understand the insane tokenization its forced to use for its vocabulary and perform accurate addition sounds great until you start trying very large numbers or messing with whatever decoding settings you're using (and let me guess, it was beam search). Or, you can simply ask it to use a tool, like a calculator for you. This is more reliable than fine-tuning a current technique BPE tokenized model ever will be.
zamadatix 3y ago
This oversimplifies too much. GPT4 is able to add random large numbers too large to have been in the data set. When it's wrong it tends to be closely wrong e.g. maybe the first 4 digits of division on a large number by a decimal number with many fractional digits are correct. Neither of these examples can be explained by memorization alone, other operational mechanics are being applied to get these outcomes.
- jjtheblunt 3y ago
  To your point, it would be really neat if we could trace what it's doing to get answers, see why it's sometimes right, sometimes close but wrong!
  - zamadatix 3y ago
    You can trace it pretty easily in a local neural net which takes a couple of minutes to train. The only reason we can't trace it in GPT is it's a closed model so we don't have access to do so. Largely two things come into play: 1) Some part of the neural net is emulating more traditional logic, but it may not always be the most activated part or tuned to be perfect in the answers 2) There isn't really a "jmp" equivalent in a single iteration, so the neural net has to learn to not only do decimal division but do it based on iterating output tokens, continuing perfectly each token output, and choosing to put the right stuff into context and keep that context activated at the right time. "Activated" in this case means, more or less, the group of neurons specializing on this task are being both fed and listened to. You can even train a neural net to emulate a traditional addition circuit directly, it's just less efficient than one would think if you're trying to build a general purpose model instead of a specialized one.

haltist 3y ago

Neural networks approximate the statistical distribution of whatever they are trained on. There is no reason to expect that any neural network will perform well on out of distribution tasks, e.g. multi-digit addition and multiplication.

ctoth 3y ago
This is (probably) wrong, see: https://gwern.net/scaling-hypothesis https://gwern.net/scaling-hypothesis And specifically the part where it discusses addition under the heading Blessings of Scale.
- haltist 3y ago
  So why can't GPT do arithmetic? At what scale do you suppose it will manage to add numbers?
  - famouswaffles 3y ago
    GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.
    - Der_Einzige 3y ago
      Besides your answer on tokenization being right on the mark (gwern even has articles discussing this exact issue - https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes), that linked paper is extremely powerful for theoretical thinking about this problem. I literally just left a comment claiming that almost all of human thought is encoded in LLMs, leading almost all tasks to be considered interpolation. If this is true, I suppose based on what we know of the curse of dimensionality it makes sense. Very thought-provoking work.
      - famouswaffles 3y ago

tsunamifury 3y ago

They aren't. This is overblown if you ask them to do simple math alone they are fine. And what they can't do they can do better with reflexion. It's not perfect, but they aren't terrible.

simbolit 3y ago
Just one (of many) examples: I asked for scores in several different categories, and the total score was not what the individiual scores added up to. I mentioned this, the model apologized, and gave another total score (also wrong).

GaggiX 3y ago

If you give them a calculator they will be much better. A better answer: every time the model needs to predict the digit of a sum the model needs to solve the entire addition to know the carries, a bigger sum requires a bigger model to solve them.

ctoth 3y ago

Tokenization, same as a year ago.

craftit 3y ago

Because often, the tokens are broken up as random groups of numbers. For example, let's say 1984 appears quite a few times in the source text, this will become a single token. Given that these many different, semi-random groups of digits it is hard for the LLM to learn any consistent rules. I believe there are papers showing that if you structure numbers more consistently LLMs have no problem with this kind of arithmetic.

ffwd 3y ago
I have a question which I don't know the answer to: With those structured numbers will the LLMs be 100% accurate on new prompts or will they just be better than chance (even significantly better than chance)? Because this is one thing, it has to learn the structure and then create probabilities based on the data, but does that mean it's actually learning the underlying algorithm for addition for example or is it just getting better probabilities because of a narrowing of them? If it can indeed learn underlying algorithms like this that's super interesting. The reason also this is in an issue if it _can't_ learn those, you can never trust the answer unless you check it, but that's sort of a sidepoint.
- craftit 3y ago
  From what I understand, it can learn and execute the algorithm fairly reliably, though it won't be 100%. When the LLM generates text, it is randomised a little, as well as some tricks that prevent repetition, which would likely cause problems with numbers containing all the same digit.

vidarh 3y ago

Why are human children so bad at math until they've repetitively trained on how to apply each basic operation over and over and over even after having learned rules that should be simple to apply? As it happens, conditioning even people to actually apply rules properly tends to take a lot of repetition. How many individual examples of step-by-step working out basic math problems do you think they have been in their training data? Prompt them the way you'd prompt a child who is sloppy into working step by step and explaining how it applies the rules it has learned, and it will tend to do better. While tokenization might not help, I don't think there's an inherent problem there beyond feeding them enough training data. Whether that's worthwhile vs. having it resort to tools, is another matter.

Verdex 3y ago

I'm definitely not on the LLM bandwagon. While I do think they're interesting, I have significant doubts that they'll be much more than a curiosity to laymen and just one more tool in the box for experts. However, I'm not sure why being bad at math (if even true based on other comments here), is a legitimate criticism. We've already got a lot of machinery that is very good at math. So, use the tools that make sense for the domain. Or (what I'm really waiting for) better yet find novel ways to stitch the tools that we do have together. Can LLMs turn a problem statement into a matlab program? If so then it doesn't really matter how bad at math they are.

spywaregorilla 3y ago

Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible.

famouswaffles 3y ago
This is wrong. The token predictor will learn algorithms for math. You can ask GPT-4 arbitrary arithmetic it could never see in it's training set. Even when it's not completely correct, it's extremely close. It is clearly computing algorithms even if those algorithms are not quite right.
- spywaregorilla 3y ago
  It is clearly not computing these algorithms. Getting close is more evidence of that than otherwise. Being able to infer answers to math problems is a thing that humans can do, and that's fine. It's not as good as doing the math though.
  - famouswaffles 3y ago
    It is computing algorithms. There's no debate here. The question is whether it is computing the right ones.
    - spywaregorilla 3y ago
      It is not computing the right ones. There is no debate there. These algorithms do not map inputs to outputs reliably.
      - famouswaffles 3y ago
        The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool or pad. The biggest problem with LLM arithmetic is tokenization. https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Other than that, the algorithms it uses for arithmetic will continue to converge during training until it is correct.