1. this is interesting. tokens are arbitrary identifiers... of text. It's called a 'text encoder' for a reason 2. it's a strong assertion but it is true. it's

1. this is interesting. tokens are arbitrary identifiers... of text. It's called a 'text encoder' for a reason 2. it's a strong assertion but it is true. it's kind of inherent in the name; next word prediction. Why would you want a calculator to be making predictions? I agree that it's worth understanding whether it's possible, but your original point was that 'arithmetic will become suddenly reliable' - this is what I am disagreeing with. 3. this links to the above point - you are right: LLM's can already perform arithmetic, this could be easily proved by loading up gpt-2. but again, your original point was that LLM's will 'figure out' arithmetic - I don't believe they will. The loss function does not incentivize being good at arithmetic, it incentivizes making good predictions that are close to the true value. The loss function will not penalize an LLM who predict 'good' is the next word when it should have been 'great'. while it might penalize '2+2=3; since this sequence would be strongly represented in the training set, it's not going to penalize the model getting '1234*5678' one digit off - which is the problem.