4 ms·
how does this approach differ from Nvidia's 2019 writeup on using trees to improve allreduce operations? https://developer.nvidia.com/blog/massively-scale-deep-
by brrrrrm 2y ago
how does this approach differ from Nvidia's 2019 writeup on using trees to improve allreduce operations? https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/ https://developer.nvidia.com/blog/massively-scale-deep-learn...
- mrfox321 2y agoSimilar, but specialized to softmax(QK)V computation