5 ms·
GPU-Oriented PCIe Expansion Cluster
- deleted 10y ago[deleted]
- EthanV2 10y agoI'm not as clued up as I used to be about this stuff, but wouldn't this have a pretty serious impact on the performance of the individual cards? Seems like splitting 4 16x cards off one 4x bus would limit the available bandwidth somewhat.
- mschuster91 10y agoThat depends if the workload is data-transfer-bound or computing-bound. If the former, yes you will suffer a massive performance blow even with just one GPU - but if the latter, it's an easy way to upgrade your system.
- EthanV2 10y agoI suppose if you're just working on a data set that's already stored in memory on the GPU(s) the initial work involved in getting that data to the card would be impacted but everything after that benefits from having an absurd amount of computing power
- rasz_pl 10y agoThis might be useful for password cracking on GPUs. I imagine there is negligent traffic on the bus in this scenario.
- minxomat 10y agoOP here. I found exactly one person using it, with positive results: https://hashcat.net/forum/thread-2622.html https://hashcat.net/forum/thread-2622.html Though amfeltec is sadly a pretty unknown company. Probably best known for their "squid" PCIe "split" cards for multiple M.2 drives in one x16 slot. I've used one of them.
- deleted 10y ago[deleted]
- StavrosK 10y agoAt those speeds, doesn't the cable/bus length play a huge role? What's the maximum length that can support them? I'd imagine it'd be a few centimeters at most...
- detaro 10y agoshorter is obviously better, but from what I've read 30 cm PCIe riser cables work, and there are 50 cm examples.
- StavrosK 10y agoHuh, that's longer than I thought, thanks.
- Radle 10y agoAs long as the insulation is good, in theory everything is possible. You could build your own super long PCI Extender Cable, using superconductors...
- throwawayish 10y agoResistive losses are only a part of the total loss on a transmission line.
- kierank 10y agoWe have problems with signal integrity on cheap risers and ribbon cables.
- deleted 10y ago[deleted]
- nabla9 10y agoGen 3 PCIe with optical cables works up to 100 meters.
- 10y ago
- mrbill 10y agoI saw a lot of this when GPUs were being used to mine bitcoin/altcoin.
- DennisP 10y agoThey're still being used for some blockchains with ASIC-resistant mining, including zcash and ethereum.
- kierank 10y agoIf there are any reasonably-priced ways of doing this especially with more PCIe lanes please let me know. In my experience it's often easier to just buy more motherboards and CPUs than invest in PCIe expansion.
- anoother 10y agoA used Tesla s1075 from eBay looks like a good bet. That's my plan, anyway. Would require some modding to use more up-to-date GPUs (there are some mounting studs on the chassis floor that need trimming) but the forced airflow means that passive cards like the original Xeon Phis are also a possibility.
- angry_octet 10y agoDepends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures: https://www.microsemi.com/products/drivers-interfaces-and-pcie-switches/pcie-switches/pcie-fanout-switches/pm8536-pfx-96xg3 https://www.microsemi.com/products/drivers-interfaces-and-pc... The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html http://www.dolphinics.com/products/IXS600.html I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders. https://events.linuxfoundation.org/sites/events/files/slides/LinuxVault2015_KeithBusch_PCIeMPath.pdf https://events.linuxfoundation.org/sites/events/files/slides...
- kierank 10y agoThe Dolphin systems seem more expensive than just buying 4U servers.
- angry_octet 10y agoYou only buy them when you can't fit your problem into one server (i.e. not enough memory or I/O) or you need redundancy (multipath I/O, synchronised memory). In those cases they can be a lot quicker/cheaper/more reliable than trying to solve it with more computers, protocols and ethernet.
- westmeal 10y agoThis is interesting but would different architectures of GPUs play nicely? E.g. 3 AMD gpus and one NVIDIA gpu?
- Kubuxu 10y agoThey should. I have currently AMD + NVIDIA GPU in my PC, the lanes are split on the motherboard (it can be 1 x16 or 2 x8). There is a bit of fiddling with drivers but it is possible to have one manufacturer working in compute only mode and other as graphics and compute.
- minxomat 10y agoTake a look at the hashcat link I posted above. Someone had success with a mixed AMD/NV setup using linux.
- throwawayish 10y agoIf you think regular GPU drivers are shitty, then wait until you try to install both AMD and nVidia drivers at the same time. It's not very stable, to say the least (Windows 7). Never tried it since. Not worth the hassle.
- dragandj 10y agoEasy to install both nvidia and AMD and works like a charm for GPGPU on Arch Linux, though.
- angry_octet 10y agoUnfortunately this setup would be very slow for most GPGPU applications, because the CPU-GPU and GPU-GPU bandwidth is very slow. To effectively use this the data transfer requirements have to be very low. Even with a dedicated 16x PCIe 3 connection there is a latency overhead compared to inter-CPU buses, like HyperTransport or QPI, which is why nVidia and IBM have scaled up the NVlink inter-GPU bus to become a memory speed interconnect. https://www.ibm.com/blogs/systems/ibm-power8-cpu-and-nvidia-pascal-gpu-speed-ahead-with-nvlink/ https://www.ibm.com/blogs/systems/ibm-power8-cpu-and-nvidia-...
- varelse 10y agoThe right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidirectional ring suitable for O(n) collectives like gather/reduce/allGather/allReduce. This is more or less useless, or if one wishes to be kind, an amazing emulation of building distributed GPU code over the craptastic bandwidths brought to you by AWS, Microsoft, and Google datacenters for now. Also PCIE Gen 2? WTF? This might not even post with current GPUs.
- chx 10y agoPCIe is all kinds of forgiving. It works all in possible combos.
- varelse 10y agoExcept that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so. Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.
- angry_octet 10y agoI'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?
- Unklejoe 10y agoA somewhat related question: Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time? Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!
- chx 10y agoThis needs a (2014) in the title.
- krautsourced 10y agoFor those doubting the usefulness of these rigs, I can say that for GPU rendering numerous of these are in use by small sized studios and freelancers. The bottleneck of the slow PCIe connection is nothing compared to the render times per frame in most cases. It's only when a frame takes less than a second to render that you notice. They are very popular with e.g. Octane users. The only thing to be aware of is that some motherboards do not play well with this and will not recognize all cards or will not run stable with it attached.