> But there's no little numeric model embedded inside the language model that would tell it how many dollars, or be able to compare dollars against dollars Thi

> But there's no little numeric model embedded inside the language model that would tell it how many dollars, or be able to compare dollars against dollars This logic applies to any function an LLM may perform, therefore it cannot perform any function, which is absurd. Indeed there functionally are little circuits inside LLMs; its pretraining amounts to formation of those circuits. [1] LLaMA tokenizer parses integers as individual tokens and allows the model to see that "$105,000" is [$][1][0][5][,][0][0][0], which makes it more than capable of learning arithmetic for the general case, and with finetuning even the smallest LLaMA-1 can learn to answer better than GPT-4.[2] It still fails a lot, but so do people, who definitely understand arithmetic on every level from practical to conceptual to algorithmic. The problem is more that it does not have a calculator inside and cannot (by default) use any explicit cache for intermediate values. 1. https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html 2. https://arxiv.org/abs/2305.14201 https://arxiv.org/abs/2305.14201