> For example, large language models and other AI systems also appear to display creativity, but they don’t harness locality and equivariance. "Next token" pre

> For example, large language models and other AI systems also appear to display creativity, but they don’t harness locality and equivariance. "Next token" prediction is (primary) local, in the sense that the early layers are largely concerned with grammatical coherence, not semantics, and if you shifted the text input context window by a few paragraphs, it would adjust the output accordingly. It's not _mathematically_ the same, but i do think the mechanics are similar.