3 ms·
Numbers aren't language, or even sequences of tokens, or vectors. There is an inherent numeric-ness and logic to math that I don't think we can represent well
by Nifty3929 1y ago
Numbers aren't language, or even sequences of tokens, or vectors.
There is an inherent numeric-ness and logic to math that I don't think we can represent well using LLMs and transformers.
3 isn't about the word "three" - it is a quantity or a measurement. And 3x4 is a specific numerical operation that is not really contained in that sequence of symbols.
- jakefromstatecs 1y agoMath is just symbol manipulation with a set of rules, no?
- Nifty3929 1y agoNo. Math and especially numbers are not just symbol manipulation. Geometry is a counter-example. So is multiplication, for that matter. Maybe you could say that algebra is just symbol manipulation. And in any case - "set of rules" is exactly what transformers aren't good at. Transformers are good at capturing the essence of what you meant and responding in a sensible, but not rule-bound way. This works well for language problems. Perhaps you could argue that transformers are just a set of rules (weights/parameters) being applied, and you might similarly argue that numbers reduce to logical symbols like S(0), S(S(0)), but then I'd argue that you're missing the point.