3 ms·

Arithmetics is extremely easy for a neural network to perform and learn perfectly, that LLMs fails to learn it even though it is so easy is strong evidence that

Arithmetics is extremely easy for a neural network to perform and learn perfectly, that LLMs fails to learn it even though it is so easy is strong evidence that LLMs has very limited capability to learn logical structures that can't be represented as grammar. > Human beings do arithmetic problems wrong all the time Humans built cars and planes and massive ships before we had calculators, that requires a massive amount of calculations that are all perfect to be possible. Humans aren't bad at getting calculations right, they are just a bit slow. Today humans are bad since we don't practice it, not because we can't. LLMs can't do that today, can learn and can't is a massive difference.

wruza 3y ago
Arithmetics is extremely easy for a neural network to perform and learn perfectly Is it?
- Jensson 3y ago
  Yes, loss minimization quickly gets to the correct implementation of arithmetics since the primitives of neural networks are just math operations, so training it to add or multiply two inputs into an output is very easy. This is so easy and obvious that you run it to test that your neural network implementation works, if it can't figure out arithmetics then you have done something wrong. LLMs fails to figure out that this is what it has to do, instead it looks like it has a ton of specialized rules to handle arithmetics that results in a lot of errors in the output and are extremely expensive to run.
  - wruza 3y ago
    So the networks you mentioned aren’t LLMs? Why is that a correct comparison then. Like blaming a human that they can’t jump like a cat or multiply like an arbitrary-precision library.
    - Jensson 3y ago
      > So the networks you mentioned aren’t LLMs? Why is that a correct comparison then Because an LLM is a neural network and neural networks contains neural networks. There is nothing stopping it from having an embedded neural network that learned how to do computations well, except an inability to identify such structures and patterns well enough to train for it.
      - jncfhnb 3y ago
        Tokenizing ‘1735’ as a value of 1735 because you’ve seen a lot of math is probably the most difficult part.
Closi 3y ago
Humans were terrible at getting calculations right - that's why we invented abacuses, slide-rules, books of mathematical tables and tabulation machines.
- Jensson 3y ago
  Humans invented those since we are slow and have limited working memory. But we managed to invent those since we understand how to perform reliable calculations.
  - Closi 3y ago
    Yes, but that acknowledges that there is a difference between understanding how to perform reliable calculations, and actually being able to perform reliable calculations. Humans are good at the former, but not the latter.
    - Jensson 3y ago
      Humans are good at performing reliable calculations with pen and paper. That is the same kind of tools that LLMs works with. I'm not sure why humans can do that but not LLMs, the task should be way easier for an LLM.
      - Closi 3y ago
        LLM's do that ok too... just not for crazy complex equations that would be tough for most humans with pen and paper. See below which I have just run on GPT4: https://chat.openai.com/share/3adb3aa2-8aec-474f-bdb0-4d761dddb5f0 https://chat.openai.com/share/3adb3aa2-8aec-474f-bdb0-4d761d...
        Jensson 3y ago
        I know they can do that, but not as reliably as I can for example or typical engineers from 80 years ago. I did engineer exams without a calculator just did all the calculations with pen and paper, didn't make mistakes, just takes a bit longer since calculating trigonometric functions takes a bit but still not a lot of time compared to how much you have. That was how everyone did it back then, it really isn't that hard to do. Most people today never tried to do it so they think it is much harder than it actually is.
pedrovhb 3y ago
My intuition is that a significant challenge for LLMs' ability to do arithmetics has to do with tokenization. For instance, `1654+73225` as per the OpenAI tokenizer tool breaks down into `165•4•+•732•25`, meaning the LLM is incapable of considering digits individually; that is, "165" is a single "word" and its relationship to "4" and in fact each other token representing a numerical value has to be learned. It can't do simple carry operations (or other arithmetic abstractions humans have access to) in the vast majority of cases because its internal representation of text is not designed for this. Arithmetic is easy to do in base 10 or 2 or 16, but it's a whole lot harder in base ~100k where 99% of the "digits" are words like "cat" or "///////////". Compare that to understanding arbitrary base64-encoded strings; that's much harder for humans to do without tools. Tokenization still isn't _the_ greatest fit for it, but it's a lot more tractable, and LLMs can do it no problem. Even understanding ASCII art is impressive, given they have no innate idea of what any letter looks like, and they "see" fragments of each letter on each line. So I'm not sure if I agree or disagree with you here. I'd say LLMs in fact have very impressive capabilities to learn logical structures. Whether grammar is the problem isn't clear to me, but their internal representation format obviously and enormously influences how much harder seemingly trivial tasks become. Perhaps some efforts in hand-tuning vocabularies could improve performance in some tasks, perhaps something different altogether is necessary, but I don't think it's an impossible hurdle to overcome.
- Closi 3y ago
  I don't think that's really how it works - sure this is true at the first level in a neural network, but in deep neural networks after the first few layers the LLM shouldn't be 'thinking' in tokens anymore. The tokens are just the input - the internal representation can be totally different (and that format isn't tokens).
  - IanCal 3y ago
    That requires converting from a weird unhelpful form into a more helpful form first, so yes but the tokenisation makes things harder as it adds an extra step - they need to learn how these things relate while having significant amounts of the structure hidden from them.
UncleEntity 3y ago
> Arithmetics is extremely easy for a neural network to perform and learn perfectly, that LLMs fails to learn it even though it is so easy is strong evidence that LLMs has very limited capability to learn logical structures that can't be represented as grammar. IDK, there was an article posted here about yet another LLM that performed very badly on the math tests because they mistakenly left out all the math training data. What impressed me was that it could learn any math at all from just 'reading' books or whatever. Though, perhaps, any correct answer could be attributed to pure luck, dunno.
HarHarVeryFunny 3y ago
> strong evidence that LLMs has very limited capability to learn logical structures that can't be represented as grammar. To add multi-digit numbers requires short term memory (are we on the units, or tens? was there a carry?), which LLMs don't have, so that's really the issue. The normal workaround for lack of memory in LLMs is "think step-by-step" to use it's own output (which gets fed back in as an input) as memory, and I'd assume that with appropriate training data and prompting an LLM could learn to do it in this fashion - not just giving the answer, but by giving all the steps. I suppose in theory LLMs could do limited precision math even without memory, if they did it in a single pass through their stack of transformer layers (96 for GPT-3) - use first few layers to add units and generate carry, next few layers to add tens, etc. I'm not sure how, or even if, one could train them to do it this way though - perhaps via a curriculum training agenda of first adding single digit numbers, then two-digit ones, etc ?
HarHarVeryFunny 3y ago
> Arithmetics is extremely easy for a neural network to perform and learn perfectly That'd depend on the design of the neural net and training objective. It's certainly not something that comes naturally to an LLM which neither has numbers as inputs or outputs, nor is trained with an arithmetic objective. Consider inputting "12345 * 10" into GPT-4. First thing it is going to do is tokenize the input, then embed these tokens, and these embedding vectors are then the starting point of what the transformer has to work with... https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You can use OpenAI's tokenizer tool (above) to see how it represents the "12345 * 10" character sequence as tokens, and the answer is that it breaks it down into the token ID sequence [4513, 1774, 353, 220, 605]. The [4513, 1774] represents the character sequence "12345", and "605" represents the character sequence "10". These token ID's will then be "embedded", which means mapping them to points in a very high dimensional space (e.g. 4096-D for LLaMA 7B), so each of those token ID's becomes a vector of 4096 1's and 0's, and these vectors are what the model itself actually sees as input. So, for "12345 * 10", what the model sees during training is that whenever it sees V1 V2 V3 V4 it should predict V5, where V1-5 are those 4096-D input token embeddings. The model has no idea what any of these mean - they might represent "the cat sat on the mat" for all it knows. They are just a bunch of token representations, and the LLM is just trying to find patterns in the examples it is given to figure out what the preferred "next token" output is. So, could you build (and train) a neural net to multiply, or add, two numbers together? Yes you could, if that is all you want to do. Is that what an LLM is? No, an LLM is a sequence predictor, not an NN designed and trained to do arithmetic, and all that is inside an LLM is a transformer (sequence-to-sequence predictor).
- Jensson 3y ago
  I know why it is hard for LLMs to learn this, that was the whole point. The way we make LLMs today means they can't identify such structures, and that is strong evidence they wont become smart just by scaling since all the things you brought up will still be true as we scale up. To solve this you would need some sub networks that are pretrained to handle numbers and math and other domains, and then you start training the giant LLM it can find and connect those things. But we don't know how to do that well yet afaik, and I bet all the big players has already tested things like that. As you say adding capabilities to the same model is hard.

wruza 3y ago

Arithmetics is extremely easy for a neural network to perform and learn perfectly Is it?

Jensson 3y ago
Yes, loss minimization quickly gets to the correct implementation of arithmetics since the primitives of neural networks are just math operations, so training it to add or multiply two inputs into an output is very easy. This is so easy and obvious that you run it to test that your neural network implementation works, if it can't figure out arithmetics then you have done something wrong. LLMs fails to figure out that this is what it has to do, instead it looks like it has a ton of specialized rules to handle arithmetics that results in a lot of errors in the output and are extremely expensive to run.
- wruza 3y ago
  So the networks you mentioned aren’t LLMs? Why is that a correct comparison then. Like blaming a human that they can’t jump like a cat or multiply like an arbitrary-precision library.
  - Jensson 3y ago
    > So the networks you mentioned aren’t LLMs? Why is that a correct comparison then Because an LLM is a neural network and neural networks contains neural networks. There is nothing stopping it from having an embedded neural network that learned how to do computations well, except an inability to identify such structures and patterns well enough to train for it.
    - jncfhnb 3y ago
      Tokenizing ‘1735’ as a value of 1735 because you’ve seen a lot of math is probably the most difficult part.

Closi 3y ago

Humans were terrible at getting calculations right - that's why we invented abacuses, slide-rules, books of mathematical tables and tabulation machines.

Jensson 3y ago
Humans invented those since we are slow and have limited working memory. But we managed to invent those since we understand how to perform reliable calculations.
- Closi 3y ago
  Yes, but that acknowledges that there is a difference between understanding how to perform reliable calculations, and actually being able to perform reliable calculations. Humans are good at the former, but not the latter.
  - Jensson 3y ago
    Humans are good at performing reliable calculations with pen and paper. That is the same kind of tools that LLMs works with. I'm not sure why humans can do that but not LLMs, the task should be way easier for an LLM.
    - Closi 3y ago
      LLM's do that ok too... just not for crazy complex equations that would be tough for most humans with pen and paper. See below which I have just run on GPT4: https://chat.openai.com/share/3adb3aa2-8aec-474f-bdb0-4d761dddb5f0 https://chat.openai.com/share/3adb3aa2-8aec-474f-bdb0-4d761d...
      - Jensson 3y ago
        I know they can do that, but not as reliably as I can for example or typical engineers from 80 years ago. I did engineer exams without a calculator just did all the calculations with pen and paper, didn't make mistakes, just takes a bit longer since calculating trigonometric functions takes a bit but still not a lot of time compared to how much you have. That was how everyone did it back then, it really isn't that hard to do. Most people today never tried to do it so they think it is much harder than it actually is.

pedrovhb 3y ago

My intuition is that a significant challenge for LLMs' ability to do arithmetics has to do with tokenization. For instance, `1654+73225` as per the OpenAI tokenizer tool breaks down into `165•4•+•732•25`, meaning the LLM is incapable of considering digits individually; that is, "165" is a single "word" and its relationship to "4" and in fact each other token representing a numerical value has to be learned. It can't do simple carry operations (or other arithmetic abstractions humans have access to) in the vast majority of cases because its internal representation of text is not designed for this. Arithmetic is easy to do in base 10 or 2 or 16, but it's a whole lot harder in base ~100k where 99% of the "digits" are words like "cat" or "///////////". Compare that to understanding arbitrary base64-encoded strings; that's much harder for humans to do without tools. Tokenization still isn't _the_ greatest fit for it, but it's a lot more tractable, and LLMs can do it no problem. Even understanding ASCII art is impressive, given they have no innate idea of what any letter looks like, and they "see" fragments of each letter on each line. So I'm not sure if I agree or disagree with you here. I'd say LLMs in fact have very impressive capabilities to learn logical structures. Whether grammar is the problem isn't clear to me, but their internal representation format obviously and enormously influences how much harder seemingly trivial tasks become. Perhaps some efforts in hand-tuning vocabularies could improve performance in some tasks, perhaps something different altogether is necessary, but I don't think it's an impossible hurdle to overcome.

Closi 3y ago
I don't think that's really how it works - sure this is true at the first level in a neural network, but in deep neural networks after the first few layers the LLM shouldn't be 'thinking' in tokens anymore. The tokens are just the input - the internal representation can be totally different (and that format isn't tokens).
- IanCal 3y ago
  That requires converting from a weird unhelpful form into a more helpful form first, so yes but the tokenisation makes things harder as it adds an extra step - they need to learn how these things relate while having significant amounts of the structure hidden from them.

UncleEntity 3y ago

> Arithmetics is extremely easy for a neural network to perform and learn perfectly, that LLMs fails to learn it even though it is so easy is strong evidence that LLMs has very limited capability to learn logical structures that can't be represented as grammar. IDK, there was an article posted here about yet another LLM that performed very badly on the math tests because they mistakenly left out all the math training data. What impressed me was that it could learn any math at all from just 'reading' books or whatever. Though, perhaps, any correct answer could be attributed to pure luck, dunno.

HarHarVeryFunny 3y ago

> strong evidence that LLMs has very limited capability to learn logical structures that can't be represented as grammar. To add multi-digit numbers requires short term memory (are we on the units, or tens? was there a carry?), which LLMs don't have, so that's really the issue. The normal workaround for lack of memory in LLMs is "think step-by-step" to use it's own output (which gets fed back in as an input) as memory, and I'd assume that with appropriate training data and prompting an LLM could learn to do it in this fashion - not just giving the answer, but by giving all the steps. I suppose in theory LLMs could do limited precision math even without memory, if they did it in a single pass through their stack of transformer layers (96 for GPT-3) - use first few layers to add units and generate carry, next few layers to add tens, etc. I'm not sure how, or even if, one could train them to do it this way though - perhaps via a curriculum training agenda of first adding single digit numbers, then two-digit ones, etc ?

HarHarVeryFunny 3y ago

> Arithmetics is extremely easy for a neural network to perform and learn perfectly That'd depend on the design of the neural net and training objective. It's certainly not something that comes naturally to an LLM which neither has numbers as inputs or outputs, nor is trained with an arithmetic objective. Consider inputting "12345 * 10" into GPT-4. First thing it is going to do is tokenize the input, then embed these tokens, and these embedding vectors are then the starting point of what the transformer has to work with... https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You can use OpenAI's tokenizer tool (above) to see how it represents the "12345 * 10" character sequence as tokens, and the answer is that it breaks it down into the token ID sequence [4513, 1774, 353, 220, 605]. The [4513, 1774] represents the character sequence "12345", and "605" represents the character sequence "10". These token ID's will then be "embedded", which means mapping them to points in a very high dimensional space (e.g. 4096-D for LLaMA 7B), so each of those token ID's becomes a vector of 4096 1's and 0's, and these vectors are what the model itself actually sees as input. So, for "12345 * 10", what the model sees during training is that whenever it sees V1 V2 V3 V4 it should predict V5, where V1-5 are those 4096-D input token embeddings. The model has no idea what any of these mean - they might represent "the cat sat on the mat" for all it knows. They are just a bunch of token representations, and the LLM is just trying to find patterns in the examples it is given to figure out what the preferred "next token" output is. So, could you build (and train) a neural net to multiply, or add, two numbers together? Yes you could, if that is all you want to do. Is that what an LLM is? No, an LLM is a sequence predictor, not an NN designed and trained to do arithmetic, and all that is inside an LLM is a transformer (sequence-to-sequence predictor).

Jensson 3y ago
I know why it is hard for LLMs to learn this, that was the whole point. The way we make LLMs today means they can't identify such structures, and that is strong evidence they wont become smart just by scaling since all the things you brought up will still be true as we scale up. To solve this you would need some sub networks that are pretrained to handle numbers and math and other domains, and then you start training the giant LLM it can find and connect those things. But we don't know how to do that well yet afaik, and I bet all the big players has already tested things like that. As you say adding capabilities to the same model is hard.