3 ms·
I halfway agree with this, but not quite. And GPT4 indeed does just fine with things like 32 * 64 in multiple different ways that humans can also easily memor
by vidarh 3y ago
I halfway agree with this, but not quite.
And GPT4 indeed does just fine with things like 32 * 64 in multiple different ways that humans can also easily memorise the rules for. When I asked it to calculate it step by step, and use easily memorisable shortcuts, it first suggested the "doubling and halving method (though it stupidly started by doubling 32 and halving 64...), and got it right.
I then told it I know the powers of two up to 2^24 by heart, and asked if that changed things.
It then reasonably pointed out this means I know 2^5=32 and 2^6 = 64, and 2^5+2^6=2^(5+6) = 2^11 = 2048 and got the rules right (that was exactly what I intended when I pointed out I remember the powers of two).
So it's not all that awful at these things. It does badly when you effectively try to get it to do maths by blind recall and without nudging it to work step by step, sure.
Where it then falls down tends to be when you ask it to do calculations which involves repetitively applying the same rules many times over, where it will tend to start out well, but occasionally make stupid little mistakes.
If anything the type of mistakes it makes are scarily close to the same kind of lapses in focus humans get when doing the same, where we just get sloppy and fail to add two numbers < 10 correctly for no good reason in the middle of doing it correctly many times, and fail to go back and verify each step.
Where some see LLMs struggling with math, I see LLMs trying to do math in a way that is disturbingly close to how a human school child would, and making the same types of mistakes.
- spywaregorilla 3y agoIt may or may not be "awful". It's probably much better than humans trying a similar math without math approach. But it's a clearly suboptimal approach. Humans and AI alike can do well with bad approaches if they must. But we can find alternative ways and we need not shoehorn AI into being LLMs.
- vidarh 3y agoWe can given them tools, just like we use tools. And I'm sure we can also find ways of "wiring" tools more directly into the models. I'm not convinced it's better than humans in general at a "math without math approach" yet, but it's certainly better than a lot of people at it. They also do make really trivial mistakes sometimes. But I also don't think there is any indication that any of this is down to things that can't easily be trained out of it. I