Interesting. The authors claim this outperforms Ring Attention for distributed computation of self-attention over multiple GPUs. Distributing computation is n

Interesting. The authors claim this outperforms Ring Attention for distributed computation of self-attention over multiple GPUs. Distributing computation is necessary whenever context is too long for self-attention's computation to fit in a single GPU's available memory. Github link: https://github.com/Zyphra/tree_attention https://github.com/Zyphra/tree_attention