Nowadays, basically no architecture with an API is using standard attention anymore. There are all kinds of attention alternatives (e.g. Hyena) and tricks (e.g.

Nowadays, basically no architecture with an API is using standard attention anymore. There are all kinds of attention alternatives (e.g. Hyena) and tricks (e.g. Sliding Window, etc.) that make this analogy, as presented, flat out incorrect. In addition, for the technical aspect to make sense, a more effective article would place the points should be shown alongside evals. For example, if you're trying to make a point about where to put important context in the prompt, show a classic needle-in-the-haystack eval, or a jacobian matrix, alongside the results. Otherwise it's largely more prompt fluff.