LLMs are compression functions. An LLM that internalizes the rules of arithmetic will compress better than one that doesn't. That improvement will be measurable

LLMs are compression functions. An LLM that internalizes the rules of arithmetic will compress better than one that doesn't. That improvement will be measurable in the loss function, as it correctly predicts more of its training data that depends on arithmetic answers to predict the next word. > Why would you want a calculator to be making predictions? If the predictions are correct, why wouldn't I? You are objecting to the entire concept of LLMs at this point, there's nothing specific to arithmetic here.