Transformers (with self-attention being the key operation) are kernel smoothers which fits easily into this view of the world. See here: http://bactra.org/noteb

Transformers (with self-attention being the key operation) are kernel smoothers which fits easily into this view of the world. See here: http://bactra.org/notebooks/nn-attention-and-transformers.html http://bactra.org/notebooks/nn-attention-and-transformers.ht...