4 ms·

The right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidir

by varelse 10y ago

The right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidirectional ring suitable for O(n) collectives like gather/reduce/allGather/allReduce. This is more or less useless, or if one wishes to be kind, an amazing emulation of building distributed GPU code over the craptastic bandwidths brought to you by AWS, Microsoft, and Google datacenters for now. Also PCIE Gen 2? WTF? This might not even post with current GPUs.

chx 10y ago
PCIe is all kinds of forgiving. It works all in possible combos.
- varelse 10y ago
  Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so. Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.
  - angry_octet 10y ago
    I'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
    - Unklejoe 10y ago
      A somewhat related question: Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time? Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!
      - angry_octet 10y ago
        The TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed. The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom. (How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)
  - chx 10y ago
    Sure, if you want performance but you said it might not POST. That's not true.
    - varelse 10y ago
      Yes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014. Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.
jackyinger 10y ago
PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...
- varelse 10y ago
  Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so. Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s). There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1. That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim. And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.
kobeya 10y ago
This is for mining or password cracking.
- varelse 10y ago
  Sure, great, but if they could build this to support Deep Learning and renegade HPC, I suspect they could sell 10,000+ units/year on the down-low. That would be $10M in revenue, maybe twice that if it also worked with Macs. NVIDIA will do their worst to shut this down because it's a direct threat to DGX-1 running neural networks that aren't entirely communication-limited (long story), but if they could throw this together, I think they could make a great quick buck before the axe falls.
  - minxomat 10y ago
    If you take a look at their clients, it's mostly massive companies that probably order large lots of OEM solutions. I don't think amfeltec needs the scale or rather isn't already at the scale you are talking about (margin, not turnover), by enterprise tax and support for OEM builds alone. The 4x splitters go for about $200. But if you want guaranteed compatibility (i.e. a full build), the price (and margin for them) will skyrocket.
  - Unklejoe 10y ago
    How could Nvidia really shut them down? It seems like they're just selling PCIe switches. I could be missing something, but I don't see anything even remotely infringing about this.
- angry_octet 10y ago
  Both bitcoin and litecoin are both far faster on ASICs. CrytoNight (Monero) is designed to be harder for ASICs but it is early days, I would expect new chips to make a significant improvement over GPUs. Since the memory requirement is only a few MB there is no need for access to large memories or a PCIe bus, and they will continue using ethernet. https://bytecoin.org/blog/proof-of-work-part-2/ https://bytecoin.org/blog/proof-of-work-part-2/
  - 10y ago

by varelse 10y ago

chx 10y ago

PCIe is all kinds of forgiving. It works all in possible combos.

varelse 10y ago
Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so. Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.
- angry_octet 10y ago
  I'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
  - Unklejoe 10y ago
    A somewhat related question: Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time? Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!
    - angry_octet 10y ago
      The TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed. The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom. (How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)
- chx 10y ago
  Sure, if you want performance but you said it might not POST. That's not true.
  - varelse 10y ago
    Yes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014. Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.

jackyinger 10y ago

PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...

varelse 10y ago
Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so. Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s). There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1. That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim. And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.

kobeya 10y ago

This is for mining or password cracking.

varelse 10y ago
Sure, great, but if they could build this to support Deep Learning and renegade HPC, I suspect they could sell 10,000+ units/year on the down-low. That would be $10M in revenue, maybe twice that if it also worked with Macs. NVIDIA will do their worst to shut this down because it's a direct threat to DGX-1 running neural networks that aren't entirely communication-limited (long story), but if they could throw this together, I think they could make a great quick buck before the axe falls.
- minxomat 10y ago
  If you take a look at their clients, it's mostly massive companies that probably order large lots of OEM solutions. I don't think amfeltec needs the scale or rather isn't already at the scale you are talking about (margin, not turnover), by enterprise tax and support for OEM builds alone. The 4x splitters go for about $200. But if you want guaranteed compatibility (i.e. a full build), the price (and margin for them) will skyrocket.
- Unklejoe 10y ago
  How could Nvidia really shut them down? It seems like they're just selling PCIe switches. I could be missing something, but I don't see anything even remotely infringing about this.
angry_octet 10y ago
Both bitcoin and litecoin are both far faster on ASICs. CrytoNight (Monero) is designed to be harder for ASICs but it is early days, I would expect new chips to make a significant improvement over GPUs. Since the memory requirement is only a few MB there is no need for access to large memories or a PCIe bus, and they will continue using ethernet. https://bytecoin.org/blog/proof-of-work-part-2/ https://bytecoin.org/blog/proof-of-work-part-2/
- 10y ago