4 ms·

Recurrence is less of issue with really large models training than it is with medium sized models. Medium sized transformer models are generally not trained wit

Recurrence is less of issue with really large models training than it is with medium sized models. Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training. And sequence parallelism is same for transformer or recurrent model. For really large models, it is in fact easier to achieve peak flops because computation required scales faster than memory bandwidth required(square vs cube).

albertzeyer 2y ago
With sequence parallelism, you mean to increase the batch size, i.e. number of sequences in a batch? > Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training Is there some word missing? You mean it's more common for large-sized Transformers? > computation required scales faster than memory bandwidth required (square vs cube) That is an interesting thought. I'm trying to understand what exactly you mean. You mean, computation time is in O(N^2) where N is the sequence length, while required memory bandwidth is in O(N^3)? Why is that?
- YetAnotherNick 2y ago
  No, it means dividing the sequence into multiple chunks and processing them one by one, very similar to recurrence. See [1]. Sequence parallelism is needed when the sequence can't fit in a single GPU. Sequence parallelism is the hardest parallelism, but it is required for longer sequence. Many models just trains for smaller sequence length for majority of the training and switch to sequence parallelism for last few percentage of training. [1]: https://arxiv.org/pdf/2105.13120 https://arxiv.org/pdf/2105.13120
  - logicchains 2y ago
    >Sequence parallelism is the hardest parallelism, but it is required for longer sequence In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).

albertzeyer 2y ago

With sequence parallelism, you mean to increase the batch size, i.e. number of sequences in a batch? > Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training Is there some word missing? You mean it's more common for large-sized Transformers? > computation required scales faster than memory bandwidth required (square vs cube) That is an interesting thought. I'm trying to understand what exactly you mean. You mean, computation time is in O(N^2) where N is the sequence length, while required memory bandwidth is in O(N^3)? Why is that?

YetAnotherNick 2y ago
No, it means dividing the sequence into multiple chunks and processing them one by one, very similar to recurrence. See [1]. Sequence parallelism is needed when the sequence can't fit in a single GPU. Sequence parallelism is the hardest parallelism, but it is required for longer sequence. Many models just trains for smaller sequence length for majority of the training and switch to sequence parallelism for last few percentage of training. [1]: https://arxiv.org/pdf/2105.13120 https://arxiv.org/pdf/2105.13120
- logicchains 2y ago
  >Sequence parallelism is the hardest parallelism, but it is required for longer sequence In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).