One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players ha

One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat. [1] https://developer.nvidia.com/nccl https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/ https://rocm.docs.amd.com/projects/rccl/en/latest/