4 ms·

I’m talking about attention only transformers. Those don’t have an autodiff but still learn. The math is actually really cool.

• lostmsu 4y ago
  > attention only transformers Can you share any good link on the subject?
  • adamnemecek 4y ago
    https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html
    • lostmsu 4y ago
      Maybe I am missing something, but I don't see any learning without autodiff.
      • adamnemecek 4y ago
        I thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2 https://arxiv.org/abs/2212.10559v2.
        • lostmsu 4y ago
          The paper speculates that it is analogous to gradient descent and empirically confirms it is similar in behavior, but it is not a rigorous proof of any kind. The momentum experiment they made also does not seem related. E.g. it just adds past values to V, which extends the effective context length.
          • adamnemecek 4y ago
            > but it is not a rigorous proof of any kind. Such is the nature of early theories.