5 ms·
Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU
by varelse 10y ago
Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so.
Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.
- angry_octet 10y agoI'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
- Unklejoe 10y agoA somewhat related question: Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time? Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!
- angry_octet 10y agoThe TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed. The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom. (How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)
- chx 10y agoSure, if you want performance but you said it might not POST. That's not true.
- varelse 10y agoYes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014. Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.