3 ms·
This is .. pretty interesting! According to the abstract, models trained long enough use some feature layers for magnitude assessment, and others for modular as
by vessenes 2y ago
This is .. pretty interesting! According to the abstract, models trained long enough use some feature layers for magnitude assessment, and others for modular assessment, (e.g. even / odd). It's surprising to me that this is a stable outcome for trained LLMs when they encounter math. Definitely not what seems simplest to my meatspace brain.
- wongarsu 2y agoMy meatspace brain can do fast accurate math up to about three digit results. After than I fall back to iterative processes with chain-of-thought, and possibly physical scratch space. My brain can however do magnitude assessment and modular assessment in near-constant time too, which I use to verify the correctness of the chain-of-thought result.
- svachalek 2y agoMeatspace brain uses digits, LLM uses tokens. i.e. when I enter "7953 + 5205" what gpt-4o is actually computing is [48186, 18, 659, 220, 26069, 20] (https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer) So saying it's not the simplest is an understatement by far, it's doing millions or billions of times as much math as a calculator would. Ask an LLM to generate a program to do math, rather than doing the math itself.
- deleted 2y ago[deleted]
- duskwuff 2y agoNor does it help that numbers are written with the most significant digit first, but actually computing the sum requires the digits to be evaluated in the opposite order.