3 ms·

- They outperform transformers at lower parameter counts. Time will tell if that hold up for more parameters - They scale linearly in complexity, which means w

- They outperform transformers at lower parameter counts. Time will tell if that hold up for more parameters - They scale linearly in complexity, which means with a longer context window they will be faster and cheaper than transformers - It's been mostly academic as far as I know, only just recently being published. I don't think there's been an opportunity to use them at 'real world scale' yet, although tbh I'm a little uncertain what you mean by it.

oersted 2y ago
It does seem to match transformers, but I wouldn't say it meaningfully outperforms them in terms of quality vs parameters. Model: #Params (M), SlimPajama (15B) ppl ↓ - GPT-3: 356M, 14.26 - Llama: 407M, 14.25 - H3: 420M, 18.23 - Mamba: 423M, 13.70 - Hyena: 435M, 17.59 - RWKV-4: 430M, 15.62 - RWKV-5: 456M, 16.53 - RWKV-6: 442M, 17.40 - RetNet: 431M, 16.23 - HGRN: 411M, 21.83 - GLA: 412M, 19.56 - HGRN2: 411M, 16.77 - xLSTM[1:0]: 409M, 13.43 - xLSTM[7:1]: 408M, 13.48 There are more detailed perplexity and task benchmarks in the paper. Overall, all the architectures perform very similarly on every benchmark, sometimes xLSTM is slightly ahead but not always, and the difference is not really meaningful. This is great news though, it means we are not losing anything by switching to xLSTM and we get important advantages like the scalable context window. I'm quite excited about this because we can potentially have the LLM remember what you say and do few-shot persistent learning from user interaction (updating "itself", the state vector). It would be very interesting if LLMs were no longer static. Although I'm sure it will be a challenge to train the model to keep such learnings in its memory long-term. The paper: https://arxiv.org/abs/2405.04517 https://arxiv.org/abs/2405.04517
- 3abiton 2y ago
  Linear scaling for context is also a bit deal. Flash attention partially solved this for TF, but xLSTM seems promising!
- piecerough 2y ago
  > It would be very interesting if LLMs were no longer static. Little bit of a nightmare too. Instructions keep piling up for you that you no longer openly can access and remove

oersted 2y ago

It does seem to match transformers, but I wouldn't say it meaningfully outperforms them in terms of quality vs parameters. Model: #Params (M), SlimPajama (15B) ppl ↓ - GPT-3: 356M, 14.26 - Llama: 407M, 14.25 - H3: 420M, 18.23 - Mamba: 423M, 13.70 - Hyena: 435M, 17.59 - RWKV-4: 430M, 15.62 - RWKV-5: 456M, 16.53 - RWKV-6: 442M, 17.40 - RetNet: 431M, 16.23 - HGRN: 411M, 21.83 - GLA: 412M, 19.56 - HGRN2: 411M, 16.77 - xLSTM[1:0]: 409M, 13.43 - xLSTM[7:1]: 408M, 13.48 There are more detailed perplexity and task benchmarks in the paper. Overall, all the architectures perform very similarly on every benchmark, sometimes xLSTM is slightly ahead but not always, and the difference is not really meaningful. This is great news though, it means we are not losing anything by switching to xLSTM and we get important advantages like the scalable context window. I'm quite excited about this because we can potentially have the LLM remember what you say and do few-shot persistent learning from user interaction (updating "itself", the state vector). It would be very interesting if LLMs were no longer static. Although I'm sure it will be a challenge to train the model to keep such learnings in its memory long-term. The paper: https://arxiv.org/abs/2405.04517 https://arxiv.org/abs/2405.04517

3abiton 2y ago
Linear scaling for context is also a bit deal. Flash attention partially solved this for TF, but xLSTM seems promising!
piecerough 2y ago
> It would be very interesting if LLMs were no longer static. Little bit of a nightmare too. Instructions keep piling up for you that you no longer openly can access and remove