3 ms·

Isn't that all of modern AI?

• immibis 2y ago
  Transformers are completely unlike RNNs.
  • tripplyons 2y ago
    There are some interesting connections between them. If you remove the softmax from the attention formula, you end up with linear attention, which has a recurrent form. I haven't read it, but the Mamba 2 paper claims to establish a stronger connection.
    • kadushka 2y ago
      * If you remove the softmax from the attention formula, you end up with linear attention* Sorry, what?
      • tripplyons 2y ago
        Here is a paper explaining it: https://arxiv.org/abs/2006.16236 https://arxiv.org/abs/2006.16236