4 ms·

The most clarifying post I've read on attention is from Cosma Shalizi[0] who points out that "Attention" is quite literally just a re-discovery/re-invention of

The most clarifying post I've read on attention is from Cosma Shalizi[0] who points out that "Attention" is quite literally just a re-discovery/re-invention of Kernel smoothing. Probably less helpful if you don't come from a quantitative background, but if you do it makes it shockingly clarifying. Once you realize this "Multi-headed Attention" is just kernel smoothing with more kernels and doing some linear transformation on the results of these (in practice: average or add)! 0. http://bactra.org/notebooks/nn-attention-and-transformers.html#attention http://bactra.org/notebooks/nn-attention-and-transformers.ht...

FreakLegion 1y ago
It's a useful realization, too, since ways of approximating kernel functions are already well-studied. Google themselves have been publishing in this area for years, e.g. https://research.google/blog/rethinking-attention-with-performers/ https://research.google/blog/rethinking-attention-with-perfo... > To resolve these issues, we introduce the Performer, a Transformer architecture with attention mechanisms that scale linearly, thus enabling faster training while allowing the model to process longer lengths, as required for certain image datasets such as ImageNet64 and text datasets such as PG-19. The Performer uses an efficient (linear) generalized attention framework, which allows a broad class of attention mechanisms based on different similarity measures (kernels). The framework is implemented by our novel Fast Attention Via Positive Orthogonal Random Features (FAVOR+) algorithm, which provides scalable low-variance and unbiased estimation of attention mechanisms that can be expressed by random feature map decompositions (in particular, regular softmax-attention). We obtain strong accuracy guarantees for this method while preserving linear space and time complexity, which can also be applied to standalone softmax operations.
- snthpy 1y ago
  So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models? 1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6 https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)
thomasahle 1y ago
For those who don't know the term "kernel smoothing", it just means ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ)) In regular attention, we let K(xᵢ, xₒ) = exp(<xᵢ, xₒ>). Note that in Attention we use K(qᵢ, kₒ) where the q (query) and k (key) vectors are not the same. Unless you define K(xᵢ, xₒ) = exp(<W_q xᵢ, W_k xₒ>) as you do in self-attention. There are also some attention mechanisms that don't use the normalization term, (∑ⱼ K(xⱼ, xₒ)), but most do.
- throwup238 1y ago
  > ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ)) That clarifies things...
  - mrks_hy 1y ago
    It does, for anybody who studied math at the level you need to understand Attention (some linear algebra). Please no low effort comments, ask if you don't have the math background and people will gladly help. This is sum notation, see https://en.m.wikipedia.org/wiki/Summation https://en.m.wikipedia.org/wiki/Summation
    - auggierose 1y ago
      No, it doesn't really clarify things. I had the best linear algebra grades in my year at my university, and if you don't know anything about kernels, this is not helpful (what are xi and yi in the first place?).
      - lambdasquirrel 1y ago
        It's all described in the referenced link. No need for everyone to get antsy. > 0. http://bactra.org/notebooks/nn-attention-and-transformers.ht http://bactra.org/notebooks/nn-attention-and-transformers.ht...
esafak 1y ago
In kernel methods the kernel is typically given, and things like positional embeddings, layer normalization, causal masking, and so on are missing. Kernel methods did not take off partly due to their computational complexity (quadratic in sample size), and transforms did precisely because they were parallelizable, and thus computationally efficient, compared with the RNNs and LSTMs that came before them. Reductions of one architecture to another are usually more enlightening from a theoretical perspective than a practical one.
3abiton 1y ago
Wow, thanks for referencing that. What a very detailed and long read!
sooheon 1y ago
Interesting read. But to me, describing attention as "a weighted mean of a continous learned k-v map lookup" is direct and descriptive, while saying "it's just kernel smoothing" is opaque and referential.
- ablob 1y ago
  Maybe take the best of both worlds: "It's just kernel smoothing, which is a weighted mean of a continuous learned kv-map lookup." Now you got the definition, and the alternative naming you can use to search for resources.
snthpy 1y ago
So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models? 1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6 https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)

FreakLegion 1y ago

It's a useful realization, too, since ways of approximating kernel functions are already well-studied. Google themselves have been publishing in this area for years, e.g. https://research.google/blog/rethinking-attention-with-performers/ https://research.google/blog/rethinking-attention-with-perfo... > To resolve these issues, we introduce the Performer, a Transformer architecture with attention mechanisms that scale linearly, thus enabling faster training while allowing the model to process longer lengths, as required for certain image datasets such as ImageNet64 and text datasets such as PG-19. The Performer uses an efficient (linear) generalized attention framework, which allows a broad class of attention mechanisms based on different similarity measures (kernels). The framework is implemented by our novel Fast Attention Via Positive Orthogonal Random Features (FAVOR+) algorithm, which provides scalable low-variance and unbiased estimation of attention mechanisms that can be expressed by random feature map decompositions (in particular, regular softmax-attention). We obtain strong accuracy guarantees for this method while preserving linear space and time complexity, which can also be applied to standalone softmax operations.

snthpy 1y ago
So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models? 1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6 https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)

thomasahle 1y ago

For those who don't know the term "kernel smoothing", it just means ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ)) In regular attention, we let K(xᵢ, xₒ) = exp(<xᵢ, xₒ>). Note that in Attention we use K(qᵢ, kₒ) where the q (query) and k (key) vectors are not the same. Unless you define K(xᵢ, xₒ) = exp(<W_q xᵢ, W_k xₒ>) as you do in self-attention. There are also some attention mechanisms that don't use the normalization term, (∑ⱼ K(xⱼ, xₒ)), but most do.

throwup238 1y ago
> ∑ᵢ yᵢ · K(xᵢ, xₒ) ⁄ (∑ⱼ K(xⱼ, xₒ)) That clarifies things...
- mrks_hy 1y ago
  It does, for anybody who studied math at the level you need to understand Attention (some linear algebra). Please no low effort comments, ask if you don't have the math background and people will gladly help. This is sum notation, see https://en.m.wikipedia.org/wiki/Summation https://en.m.wikipedia.org/wiki/Summation
  - auggierose 1y ago
    No, it doesn't really clarify things. I had the best linear algebra grades in my year at my university, and if you don't know anything about kernels, this is not helpful (what are xi and yi in the first place?).
    - lambdasquirrel 1y ago
      It's all described in the referenced link. No need for everyone to get antsy. > 0. http://bactra.org/notebooks/nn-attention-and-transformers.ht http://bactra.org/notebooks/nn-attention-and-transformers.ht...

esafak 1y ago

In kernel methods the kernel is typically given, and things like positional embeddings, layer normalization, causal masking, and so on are missing. Kernel methods did not take off partly due to their computational complexity (quadratic in sample size), and transforms did precisely because they were parallelizable, and thus computationally efficient, compared with the RNNs and LSTMs that came before them. Reductions of one architecture to another are usually more enlightening from a theoretical perspective than a practical one.

3abiton 1y ago

Wow, thanks for referencing that. What a very detailed and long read!

sooheon 1y ago

Interesting read. But to me, describing attention as "a weighted mean of a continous learned k-v map lookup" is direct and descriptive, while saying "it's just kernel smoothing" is opaque and referential.

ablob 1y ago
Maybe take the best of both worlds: "It's just kernel smoothing, which is a weighted mean of a continuous learned kv-map lookup." Now you got the definition, and the alternative naming you can use to search for resources.

snthpy 1y ago

So do we think that some form of this is what they are using internally to get those long context lengths by already using sub-quadratic architectures [1] in their deployed models? 1: "Context Is The Next Frontier by Jacob Buckman, CEO of Manifest AI" (https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6 https://youtu.be/wJyl6kBCwmY?si=ruxMWdENjazu3rp6)