6 ms·

Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memoriz

Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible.

famouswaffles 3y ago
This is wrong. The token predictor will learn algorithms for math. You can ask GPT-4 arbitrary arithmetic it could never see in it's training set. Even when it's not completely correct, it's extremely close. It is clearly computing algorithms even if those algorithms are not quite right.
- spywaregorilla 3y ago
  It is clearly not computing these algorithms. Getting close is more evidence of that than otherwise. Being able to infer answers to math problems is a thing that humans can do, and that's fine. It's not as good as doing the math though.
  - famouswaffles 3y ago
    It is computing algorithms. There's no debate here. The question is whether it is computing the right ones.
    - spywaregorilla 3y ago
      It is not computing the right ones. There is no debate there. These algorithms do not map inputs to outputs reliably.
      - famouswaffles 3y ago
        The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool or pad. The biggest problem with LLM arithmetic is tokenization. https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Other than that, the algorithms it uses for arithmetic will continue to converge during training until it is correct.
        spywaregorilla 3y ago
        I'm not claiming it's about memorization. I have not memorized 9 x 9 + 1 is 82. I can simply do it without invoking any calculation. That is a close equivalent to trying to do with math token prediction. Humans doing things "without a pad" is not the same thing as doing mental math vs. trying to intuit an answer. Token predictions will not converge on correct arithmetic algorithms.
        famouswaffles 3y ago
        >I'm not claiming it's about memorization. I have not memorized 9 x 9 + 1 is 82. I can simply do it without invoking any calculation. >What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation Performing math how you suggest is little more than memorization except you've just memorized a different chain in the process. >That is a close equivalent to trying to do with math token prediction. This isn't going to become true no matter how much you repeat it. You've consistently made assertions that are trivially proven false. >Token predictions will not converge on correct arithmetic algorithms. Yes it will. It literally will. This isn't some debate. This is something that has been researched. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... You think you have an understanding of Language Models and token prediction. Unfortunately you don't.
criddell 3y ago
If you asked GPT to write a program that multiplies 32 and 64 it could do it. Since the thing is a computer, why can’t it answer queries by writing and executing a program? Does a transformer-based AI have to be 100% transformer?
- samus 3y ago
  This is simply not how transformer models work. Perhaps overly simplified, it maps an input string to an output string by pushing it through a neural network. But enabling it to access databases, APIs and to write and execute programs on accellerators would indeed significantly increase its capabilities.
- spywaregorilla 3y ago
  Of course that's ok. That's also how brains work, delegating certain tasks to specific specialized parts and processes within the brain. But it's no longer a token prediction model.
  - criddell 3y ago
    If GPT transforms are Turing-complete[1], then why couldn't the ability to write and execute a program be part of a prediction model? [1]: https://aclanthology.org/2020.conll-1.37.pdf https://aclanthology.org/2020.conll-1.37.pdf
    - spywaregorilla 3y ago
      Because the optimization logic of tweaking the gpt transforms via training data and nudging weights and biases to achieve token prediction is not a path to utilizing the full potential of the gpt transforms.
      - criddell 3y ago
        I don’t see why not. I haven’t tried this, but I think I could describe a very basic logical CPU to it and have it execute opcodes and tell me the CPU state at each step (like Knuth’s MIX). Couldn’t I ask it to write programs and execute programs for this computer when doing so would help it give better answers?
vidarh 3y ago
I halfway agree with this, but not quite. And GPT4 indeed does just fine with things like 32 * 64 in multiple different ways that humans can also easily memorise the rules for. When I asked it to calculate it step by step, and use easily memorisable shortcuts, it first suggested the "doubling and halving method (though it stupidly started by doubling 32 and halving 64...), and got it right. I then told it I know the powers of two up to 2^24 by heart, and asked if that changed things. It then reasonably pointed out this means I know 2^5=32 and 2^6 = 64, and 2^5+2^6=2^(5+6) = 2^11 = 2048 and got the rules right (that was exactly what I intended when I pointed out I remember the powers of two). So it's not all that awful at these things. It does badly when you effectively try to get it to do maths by blind recall and without nudging it to work step by step, sure. Where it then falls down tends to be when you ask it to do calculations which involves repetitively applying the same rules many times over, where it will tend to start out well, but occasionally make stupid little mistakes. If anything the type of mistakes it makes are scarily close to the same kind of lapses in focus humans get when doing the same, where we just get sloppy and fail to add two numbers < 10 correctly for no good reason in the middle of doing it correctly many times, and fail to go back and verify each step. Where some see LLMs struggling with math, I see LLMs trying to do math in a way that is disturbingly close to how a human school child would, and making the same types of mistakes.
- spywaregorilla 3y ago
  It may or may not be "awful". It's probably much better than humans trying a similar math without math approach. But it's a clearly suboptimal approach. Humans and AI alike can do well with bad approaches if they must. But we can find alternative ways and we need not shoehorn AI into being LLMs.
  - vidarh 3y ago
    We can given them tools, just like we use tools. And I'm sure we can also find ways of "wiring" tools more directly into the models. I'm not convinced it's better than humans in general at a "math without math approach" yet, but it's certainly better than a lot of people at it. They also do make really trivial mistakes sometimes. But I also don't think there is any indication that any of this is down to things that can't easily be trained out of it. I

famouswaffles 3y ago

This is wrong. The token predictor will learn algorithms for math. You can ask GPT-4 arbitrary arithmetic it could never see in it's training set. Even when it's not completely correct, it's extremely close. It is clearly computing algorithms even if those algorithms are not quite right.

spywaregorilla 3y ago
It is clearly not computing these algorithms. Getting close is more evidence of that than otherwise. Being able to infer answers to math problems is a thing that humans can do, and that's fine. It's not as good as doing the math though.
- famouswaffles 3y ago
  It is computing algorithms. There's no debate here. The question is whether it is computing the right ones.
  - spywaregorilla 3y ago
    It is not computing the right ones. There is no debate there. These algorithms do not map inputs to outputs reliably.
    - famouswaffles 3y ago
      The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool or pad. The biggest problem with LLM arithmetic is tokenization. https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Other than that, the algorithms it uses for arithmetic will continue to converge during training until it is correct.
      - spywaregorilla 3y ago
        I'm not claiming it's about memorization. I have not memorized 9 x 9 + 1 is 82. I can simply do it without invoking any calculation. That is a close equivalent to trying to do with math token prediction. Humans doing things "without a pad" is not the same thing as doing mental math vs. trying to intuit an answer. Token predictions will not converge on correct arithmetic algorithms.
        famouswaffles 3y ago
        >I'm not claiming it's about memorization. I have not memorized 9 x 9 + 1 is 82. I can simply do it without invoking any calculation. >What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation Performing math how you suggest is little more than memorization except you've just memorized a different chain in the process. >That is a close equivalent to trying to do with math token prediction. This isn't going to become true no matter how much you repeat it. You've consistently made assertions that are trivially proven false. >Token predictions will not converge on correct arithmetic algorithms. Yes it will. It literally will. This isn't some debate. This is something that has been researched. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... You think you have an understanding of Language Models and token prediction. Unfortunately you don't.

criddell 3y ago

If you asked GPT to write a program that multiplies 32 and 64 it could do it. Since the thing is a computer, why can’t it answer queries by writing and executing a program? Does a transformer-based AI have to be 100% transformer?

samus 3y ago
This is simply not how transformer models work. Perhaps overly simplified, it maps an input string to an output string by pushing it through a neural network. But enabling it to access databases, APIs and to write and execute programs on accellerators would indeed significantly increase its capabilities.
spywaregorilla 3y ago
Of course that's ok. That's also how brains work, delegating certain tasks to specific specialized parts and processes within the brain. But it's no longer a token prediction model.
- criddell 3y ago
  If GPT transforms are Turing-complete[1], then why couldn't the ability to write and execute a program be part of a prediction model? [1]: https://aclanthology.org/2020.conll-1.37.pdf https://aclanthology.org/2020.conll-1.37.pdf
  - spywaregorilla 3y ago
    Because the optimization logic of tweaking the gpt transforms via training data and nudging weights and biases to achieve token prediction is not a path to utilizing the full potential of the gpt transforms.
    - criddell 3y ago
      I don’t see why not. I haven’t tried this, but I think I could describe a very basic logical CPU to it and have it execute opcodes and tell me the CPU state at each step (like Knuth’s MIX). Couldn’t I ask it to write programs and execute programs for this computer when doing so would help it give better answers?

vidarh 3y ago

I halfway agree with this, but not quite. And GPT4 indeed does just fine with things like 32 * 64 in multiple different ways that humans can also easily memorise the rules for. When I asked it to calculate it step by step, and use easily memorisable shortcuts, it first suggested the "doubling and halving method (though it stupidly started by doubling 32 and halving 64...), and got it right. I then told it I know the powers of two up to 2^24 by heart, and asked if that changed things. It then reasonably pointed out this means I know 2^5=32 and 2^6 = 64, and 2^5+2^6=2^(5+6) = 2^11 = 2048 and got the rules right (that was exactly what I intended when I pointed out I remember the powers of two). So it's not all that awful at these things. It does badly when you effectively try to get it to do maths by blind recall and without nudging it to work step by step, sure. Where it then falls down tends to be when you ask it to do calculations which involves repetitively applying the same rules many times over, where it will tend to start out well, but occasionally make stupid little mistakes. If anything the type of mistakes it makes are scarily close to the same kind of lapses in focus humans get when doing the same, where we just get sloppy and fail to add two numbers < 10 correctly for no good reason in the middle of doing it correctly many times, and fail to go back and verify each step. Where some see LLMs struggling with math, I see LLMs trying to do math in a way that is disturbingly close to how a human school child would, and making the same types of mistakes.

spywaregorilla 3y ago
It may or may not be "awful". It's probably much better than humans trying a similar math without math approach. But it's a clearly suboptimal approach. Humans and AI alike can do well with bad approaches if they must. But we can find alternative ways and we need not shoehorn AI into being LLMs.
- vidarh 3y ago
  We can given them tools, just like we use tools. And I'm sure we can also find ways of "wiring" tools more directly into the models. I'm not convinced it's better than humans in general at a "math without math approach" yet, but it's certainly better than a lot of people at it. They also do make really trivial mistakes sometimes. But I also don't think there is any indication that any of this is down to things that can't easily be trained out of it. I