3 ms·

From the paper, seems like they are using RDMA to/from video cards, skipping the nic. > * These transactions require GPU-to-RDMA NIC support for optimal perfor

From the paper, seems like they are using RDMA to/from video cards, skipping the nic. > * These transactions require GPU-to-RDMA NIC support for optimal performance* Remarkably consumer computing actually has similarly found reason to bypass sending data through the cpu; texture streaming. DirectStorage and Sony's Kraken purport to let the GPU read direct from the SSD. It's a storage application instead of NIC, but still built around PCIe DMA-P2P (at least the DirectStorage is I think). Table 2, network stats for 128 GPUs is kind of interesting. Most topologies such as AllGather and AllReduce run with only 4 Queue Pairs. Not my area of expertise at all but wow that seems tiny! All this network, and basically everyone's talking to only a few peers? That's what it means right? The discussion at the end of the paper talked about Flowlets. The description makes me think a little bit of hash bucket chaining, where you try the first path, and if latter a conflict arise or the oath degrades, there's a fallback path already planned. Like there's would be a fallback chained bucket in a hash.

wmf 2y ago
The NIC is still there but they're skipping the data copy from system RAM to GPU RAM. https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect
mrlongroots 2y ago
> still built around PCIe DMA-P2P Right! GPUs are bandwidth behemoths. If you have a 400 Gbit/s network and a GPU that can saturate it, there's a very good bandwidth reason to cut the CPU out of the path. For collectives, you are probably optimizing for latency vs bandwidth, but in all cases the less you move the data the better it is. > Most topologies such as AllGather and AllReduce run with only 4 Queue Pairs. Not my area of expertise at all but wow that seems tiny! All this network, and basically everyone's talking to only a few peers? That's what it means right? 1. It seems that the cardinality of the collective groups is quite small (~128 GPUs), despite the reported cluster being 16k GPUs. They attribute it to "multidimensional parallelism" and while I don't speak much ML, I'm guessing that it's some sort of horizontal + vertical partitioning, and collectives are only triggered within a partition. 2. As these are collectives, all nodes do participate, but you create an overlay network with a ring or tree topology per collective. In a ring, every node just needs a QP each for left and right neighbors. For trees, it depends on the fanout. Now you could have shallow trees with larger fanouts, as latency is a function of the depth, and I feel like you can have more than 4 QPs (~500 is where NIC SRAM becomes an issue AFAIK). I think there is some buffering overhead per QP (beyond the standard NIC state) that they are trying to minimize, haven't read the paper thoroughly. 3. You do need a fancy network for collectives. They are a latency-sensitive operation on the critical path. The entire multimillion dollar thing is idle while an AllReduce is active, so 500us vs 100us in E2E latency matters a lot.
latchkey 2y ago
Both Intel (Gaudi 3) and Tenstorrent (Wormhole) have built NIC's into the GPUs. I think that we are going to see more of that in the future, but the only problem with this is that your failure cases become more extreme. NIC dies, I swap a NIC... GPUNIC dies... I swap both. 128 NICs/GPUs is the largest single non-blocking 400G switch size today using a broadcom t5 chip [0]. If you want to go larger than that, you have to go with 6x switches in a spine/leaf configuration, which greatly increases the costs. We're (Hot Aisle) in the process of deploying a cluster of 128 MI300x right now with this configuration using a Dell Z9864F switch [1]. https://hotaisle.xyz/networking/ https://hotaisle.xyz/networking/ and https://hotaisle.xyz/compute/ https://hotaisle.xyz/compute/ [0] https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm78900-series https://www.broadcom.com/products/ethernet-connectivity/swit... [1] https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/dell-powerswitch-z9864f-on-spec-sheet.pdf https://www.delltechnologies.com/asset/en-us/products/networ...

wmf 2y ago

The NIC is still there but they're skipping the data copy from system RAM to GPU RAM. https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect

mrlongroots 2y ago

> still built around PCIe DMA-P2P Right! GPUs are bandwidth behemoths. If you have a 400 Gbit/s network and a GPU that can saturate it, there's a very good bandwidth reason to cut the CPU out of the path. For collectives, you are probably optimizing for latency vs bandwidth, but in all cases the less you move the data the better it is. > Most topologies such as AllGather and AllReduce run with only 4 Queue Pairs. Not my area of expertise at all but wow that seems tiny! All this network, and basically everyone's talking to only a few peers? That's what it means right? 1. It seems that the cardinality of the collective groups is quite small (~128 GPUs), despite the reported cluster being 16k GPUs. They attribute it to "multidimensional parallelism" and while I don't speak much ML, I'm guessing that it's some sort of horizontal + vertical partitioning, and collectives are only triggered within a partition. 2. As these are collectives, all nodes do participate, but you create an overlay network with a ring or tree topology per collective. In a ring, every node just needs a QP each for left and right neighbors. For trees, it depends on the fanout. Now you could have shallow trees with larger fanouts, as latency is a function of the depth, and I feel like you can have more than 4 QPs (~500 is where NIC SRAM becomes an issue AFAIK). I think there is some buffering overhead per QP (beyond the standard NIC state) that they are trying to minimize, haven't read the paper thoroughly. 3. You do need a fancy network for collectives. They are a latency-sensitive operation on the critical path. The entire multimillion dollar thing is idle while an AllReduce is active, so 500us vs 100us in E2E latency matters a lot.

latchkey 2y ago

Both Intel (Gaudi 3) and Tenstorrent (Wormhole) have built NIC's into the GPUs. I think that we are going to see more of that in the future, but the only problem with this is that your failure cases become more extreme. NIC dies, I swap a NIC... GPUNIC dies... I swap both. 128 NICs/GPUs is the largest single non-blocking 400G switch size today using a broadcom t5 chip [0]. If you want to go larger than that, you have to go with 6x switches in a spine/leaf configuration, which greatly increases the costs. We're (Hot Aisle) in the process of deploying a cluster of 128 MI300x right now with this configuration using a Dell Z9864F switch [1]. https://hotaisle.xyz/networking/ https://hotaisle.xyz/networking/ and https://hotaisle.xyz/compute/ https://hotaisle.xyz/compute/ [0] https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm78900-series https://www.broadcom.com/products/ethernet-connectivity/swit... [1] https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/dell-powerswitch-z9864f-on-spec-sheet.pdf https://www.delltechnologies.com/asset/en-us/products/networ...