2 ms·

PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...

varelse 10y ago
Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so. Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s). There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1. That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim. And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.

2 ms·

PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...

by jackyinger 10y ago

varelse 10y ago
Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so. Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s). There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1. That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim. And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.