That's an inductive bias, not an emergent property. The inductive bias of transformers is that they're good at integrating global context from different parts o

That's an inductive bias, not an emergent property. The inductive bias of transformers is that they're good at integrating global context from different parts of a sequence without a particular bias towards recent time steps or localized regularities. And that happens to be a good fit for many (but not all) real-world sequence learning tasks. The "emergent property" aspect is when LLMs are good at a task at scale X*3 but were incompetent at scale X.