The paper is about the ability of transformers to learn a task based on training data for that task only, not about LLMs pretrained on much of the internet. And

The paper is about the ability of transformers to learn a task based on training data for that task only, not about LLMs pretrained on much of the internet. And training on log tables doesn't necessarily allow the model to always output the correct logarithm, just as training on multiplication tables doesn't necessarily confer the ability to multiply.