4 ms·

It seems Sepp Hochreiter has talked already about this model since Oct 2023: https://github.com/huggingface/transformers/issues/27011 https://github.com/hugging

It seems Sepp Hochreiter has talked already about this model since Oct 2023: https://github.com/huggingface/transformers/issues/27011 https://github.com/huggingface/transformers/issues/27011 In the scaling law comparison, I wonder if it is reasonable to compare number of parameters between Llama, Mamba, RWKV, xLSTM? Isn't compute time more relevant? E.g. in the figure about scaling laws, replace num of params by compute time. Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation. So scaling up Transformer could still look better when you look at compute time. It seems neither the code nor the model params are released. I wonder if that will follow.

YetAnotherNick 2y ago
Recurrence is less of issue with really large models training than it is with medium sized models. Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training. And sequence parallelism is same for transformer or recurrent model. For really large models, it is in fact easier to achieve peak flops because computation required scales faster than memory bandwidth required(square vs cube).
- albertzeyer 2y ago
  With sequence parallelism, you mean to increase the batch size, i.e. number of sequences in a batch? > Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training Is there some word missing? You mean it's more common for large-sized Transformers? > computation required scales faster than memory bandwidth required (square vs cube) That is an interesting thought. I'm trying to understand what exactly you mean. You mean, computation time is in O(N^2) where N is the sequence length, while required memory bandwidth is in O(N^3)? Why is that?
  - YetAnotherNick 2y ago
    No, it means dividing the sequence into multiple chunks and processing them one by one, very similar to recurrence. See [1]. Sequence parallelism is needed when the sequence can't fit in a single GPU. Sequence parallelism is the hardest parallelism, but it is required for longer sequence. Many models just trains for smaller sequence length for majority of the training and switch to sequence parallelism for last few percentage of training. [1]: https://arxiv.org/pdf/2105.13120 https://arxiv.org/pdf/2105.13120
    - logicchains 2y ago
      >Sequence parallelism is the hardest parallelism, but it is required for longer sequence In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).
korbip 2y ago
Disclaimer: I'm shared first author of this paper. As a clarification: The speed for training will be on par with FlashAttention-2, when fully optimized and only including the mLSTM. For decoding/inference both are very close to Mamba as xLSTM is a recurrent architecture. The sLSTM has memory mixing, that is state tracking capabilities, for problems Transformers and State Space Models (and any other sequence-parallelizable architecture) cannot solve fundamentally.
- albertzeyer 2y ago
  Congratulations on the paper. That's some very interesting work! But you would want to include sLSTM as well to get the best performance, right? How does the speed compares in that case? Specifically when scaling up.
  - korbip 2y ago
    Thank you! I can say that it is not really a diminishing factor at the scales reported in the paper. So, xLSTM[7:1] is pretty much on par with xLSTM[1:0] in speed. We show that it is helpful on toy tasks, and it shows even better sequence extrapolation performance, so yes.
- WithinReason 2y ago
  Can you expand on the "cannot solve fundamentally" part?
  - lucidrains 2y ago
    https://arxiv.org/abs/2404.08819 https://arxiv.org/abs/2404.08819
    - deleted 2y ago
      [deleted]
    - Der_Einzige 2y ago
      So does anything do proper state tracking? And don’t point to the OP since very often purportedly better new architectures end up being basically vaporware (like mamba or rkwv, which still don’t have good quality pre trained models yet)
zozbot234 2y ago
> Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation. If you mean that you cannot fully parallelize inference, this might be true but also not quite relevant since the computational demands of inference are low. And you can always "parallelize" training to some extent, just by training larger batches.
- korbip 2y ago
  This was formulated a bit unclear. It is not possible to parallelize in the sequence dimension for training as it is possible for Transformers. In the batch dimension you can always do it.

YetAnotherNick 2y ago

Recurrence is less of issue with really large models training than it is with medium sized models. Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training. And sequence parallelism is same for transformer or recurrent model. For really large models, it is in fact easier to achieve peak flops because computation required scales faster than memory bandwidth required(square vs cube).

albertzeyer 2y ago
With sequence parallelism, you mean to increase the batch size, i.e. number of sequences in a batch? > Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training Is there some word missing? You mean it's more common for large-sized Transformers? > computation required scales faster than memory bandwidth required (square vs cube) That is an interesting thought. I'm trying to understand what exactly you mean. You mean, computation time is in O(N^2) where N is the sequence length, while required memory bandwidth is in O(N^3)? Why is that?
- YetAnotherNick 2y ago
  No, it means dividing the sequence into multiple chunks and processing them one by one, very similar to recurrence. See [1]. Sequence parallelism is needed when the sequence can't fit in a single GPU. Sequence parallelism is the hardest parallelism, but it is required for longer sequence. Many models just trains for smaller sequence length for majority of the training and switch to sequence parallelism for last few percentage of training. [1]: https://arxiv.org/pdf/2105.13120 https://arxiv.org/pdf/2105.13120
  - logicchains 2y ago
    >Sequence parallelism is the hardest parallelism, but it is required for longer sequence In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).

korbip 2y ago

Disclaimer: I'm shared first author of this paper. As a clarification: The speed for training will be on par with FlashAttention-2, when fully optimized and only including the mLSTM. For decoding/inference both are very close to Mamba as xLSTM is a recurrent architecture. The sLSTM has memory mixing, that is state tracking capabilities, for problems Transformers and State Space Models (and any other sequence-parallelizable architecture) cannot solve fundamentally.

albertzeyer 2y ago
Congratulations on the paper. That's some very interesting work! But you would want to include sLSTM as well to get the best performance, right? How does the speed compares in that case? Specifically when scaling up.
- korbip 2y ago
  Thank you! I can say that it is not really a diminishing factor at the scales reported in the paper. So, xLSTM[7:1] is pretty much on par with xLSTM[1:0] in speed. We show that it is helpful on toy tasks, and it shows even better sequence extrapolation performance, so yes.
WithinReason 2y ago
Can you expand on the "cannot solve fundamentally" part?
- lucidrains 2y ago
  https://arxiv.org/abs/2404.08819 https://arxiv.org/abs/2404.08819
  - deleted 2y ago
    [deleted]
  - Der_Einzige 2y ago
    So does anything do proper state tracking? And don’t point to the OP since very often purportedly better new architectures end up being basically vaporware (like mamba or rkwv, which still don’t have good quality pre trained models yet)

zozbot234 2y ago

> Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation. If you mean that you cannot fully parallelize inference, this might be true but also not quite relevant since the computational demands of inference are low. And you can always "parallelize" training to some extent, just by training larger batches.

korbip 2y ago
This was formulated a bit unclear. It is not possible to parallelize in the sequence dimension for training as it is possible for Transformers. In the batch dimension you can always do it.