3 ms·

for those who want a more conversational intro, we've been covering RWKV for a bit! 2023: https://latent.space/p/rwkv https://latent.space/p/rwkv 2024: https:

• inciampati 2y ago
  the recurrent model needs a mechanism to replay past context. no need to go quadratic to access all of it. they could replay multiple times to get effects similar to attention. the hardware lottery, well... imo it's really about leveraging fully parallel training to learn how to use a memory. attention is quadratic but it can be computed in parallel. it's an end to end learned memory. getting that kind of pattern into RNNs won't be easy but it's going to be crucial before we boil the ocean.
  • pico_creator 2y ago
    RWKV already solve the parallel compute problem for GPU, based on the changes it has done - so it is a recurrent model that can scale to thousands++ of GPU no issue. Arguably with other recurrent architecture (State Space, etc) with very different design implementation. The issue of old recurrent design was just the way LSTM was designed. Not the recurrent nature.
• shawntan 2y ago
  Although marketed as such, RWKV isn't really an RNN. In the recent RWKV7 incarnation, you could argue it's a type of Linear RNN, but past versions had an issue of taking its previous state from a lower layer, allowing for parallelism, but makes it closer to a convolution than a recurrent computation. As for 1), I'd like to believe so, but it's hard to get people away from the addictive drug that is the easily parallelised transformer, 2) (actual) RNNs and attention mechanisms to me seem fairly powerful (expressivity wise) and perhaps most acceptable by the community.
  • bravura 2y ago
    Recent work by Feng et al from Bengio's lab focus on how attention can be formulated as an RNN ("Attention as RNN": https://arxiv.org/pdf/2405.13956 https://arxiv.org/pdf/2405.13956) and how minimal versions of GRUs and LSTMs can be trained in parallel by removing some parameters ("Were RNNs All We Needed?" https://arxiv.org/pdf/2410.01201 https://arxiv.org/pdf/2410.01201). It's possible we start seeing more blended version of RNN/attention architecture exploring different LLM properties. In particular, Aaren architecture in the former paper "can not only (i) be trained in parallel (like Transformers) but also (ii) be updated efficiently with new tokens, requiring only constant memory for inferences (like traditional RNNs)."
    • shawntan 2y ago
      The formulations in attention as rnn have similar issues as rwkv. Fundamentally it's a question of what we call an RNN. Personally I think it's important not to call some of these recent architectures RNNs because they have theoretical properties that do not match (read: they're worse) what we've "classically" called RNNs. Ref: https://arxiv.org/abs/2404.08819 https://arxiv.org/abs/2404.08819 As a rule of thumb: you generally don't get parallelism for free, you pay for it with poorer expressivity.
• HarHarVeryFunny 2y ago
  The Transformer was specifically conceived to take advantage of pre-existing massively parallel hardware, so it's a bit backwards to say it "won the hardware lottery". Where the Transformer did "win the lottery" is that the key-value form of self-attention (invented by Noam Shazeer) needed to make parallel processing work seems to have accidentally unlocked capabilities like "induction heads" that make this type of architecture extremely well suited to language prediction. Given limits on clock speed, massive parallelism is always going to be the way to approach brain-like levels of parallel computation, so any model architecture aspiring to human level AGI needs to be able to take advantage of that.
  • swyx 2y ago
    you are correct of course but i meant hardware lottery in the sense of dedicated silicon companies like Etched and MatX that have now emerged to make chips that only run transformers (not exactly true for matx but hey i am simplifying. would be cool if matx ran other arch's but its not a priority)