5 ms·
I've been evaluating this space a fair bit recently. If you want to optimize FLOPS/$, especially for research-workstation sorts of setups, there's unfortunately
by highd 9y ago
I've been evaluating this space a fair bit recently. If you want to optimize FLOPS/$, especially for research-workstation sorts of setups, there's unfortunately not a lot of options for getting more than 6 GPUs in a motherboard without going to server grade where you're spending basically $3-4K more for unclear benefit - maybe a factor of two in GPU-to-GPU bandwidth.
The bitcoin miners have figured out one way to handle this, which is by using a variety of PCIe splitting systems. I've seen examples of people putting in 8GPUs in 4 slots with these splitters. The problem is that the majority of these splitters take your x16 connection and turn into 2 to 4 x1 PCIe lanes, which is a lot of wasted bandwidth. This is fine for the miners, since the cards run mostly independently. If I could find compatible PCIe splitters that could split x16 into 2 x8 channels then that would be a really sweet spot in performance/$, but unfortunately I've yet to find them. So right now I'm going to stick to 6 GPUs, which you can get in a $500 consumer motherboard with just a few riser cables.
See for example: http://amfeltec.com/products/flexible-x4-pci-express-4-way-splitter-gpu-oriented/ http://amfeltec.com/products/flexible-x4-pci-express-4-way-s...
- forgotmyhnacc 9y agoWhat about AMDs threadripper? It supports 60 PCIe lanes.
- wmf 9y agoBut what slots will the motherboards have? Four x16 or eight x8?
- edaemon 9y agoThe ones I've seen so far have either four or five slots. The lane allocation isn't set in stone; I've seen a few that have 3-x16, 1-x1, and 3-x4 (M2) [1] but I've also seen one that has 4-x16, 1-x1, and 2-x4 (M2) [2]. I assume those motherboards are using PCIe switches, but there are plenty of configurations that wouldn't require the switch. [1] https://www.pcper.com/news/Motherboards/Computex-2017-ASRock-Shows-Two-X399-Threadripper-Motherboards https://www.pcper.com/news/Motherboards/Computex-2017-ASRock... [2] http://wccftech.com/gigabyte-x399-aorus-gaming-7-amd-ryzen-threadripper-motherboard-pictured/ http://wccftech.com/gigabyte-x399-aorus-gaming-7-amd-ryzen-t...
- deleted 9y ago[deleted]
- brigade 9y agoNot sure if you already knew about them, but there are consumer X99 motherboards that include PLX8747 switches to mux 32 PCIe lanes from the CPU into 7 PCIe x8 slots, at premium of maybe $200 for the motherboard and $300 for a compatible CPU. (ASUS X99-E-10G WS) The catch being that the 7 slots are next to each other, so you either have to make a custom water loop with single slot GPUs or use simple risers on half of them. But that's probably the best bandwidth you can currently get between >4 GPUs on consumer parts.
- highd 9y agoThat sounds amazing - do you know of any in particular? I can't seem to find any motherboards with 8 PCIe slots. Or do they need some sort of additional splitter?
- brigade 9y agoSorry, I was wrong, it's only 7 slots - ASUS x99-e-10g ws is the board I was thinking of.
- highd 9y agoStill pretty good - thanks!
- deleted 9y ago[deleted]
- mikepurvis 9y agoPretty amazing, though— there's a picture of it on this review: http://proclockers.com/reviews/motherboards/asrock-x99-ws-e-motherboard-review?nopaging=1 http://proclockers.com/reviews/motherboards/asrock-x99-ws-e-...
- Keyframe 9y agoHere's a guy building a water-cooled 7x1080 system https://www.youtube.com/watch?v=9hsQmcSwGv0&list=PLj4jtAjLgQsJDVcdoVncbYu6BTrFetgz0 https://www.youtube.com/watch?v=9hsQmcSwGv0&list=PLj4jtAjLgQ... With Ti series you don't have to cut the excess from the card either it seems, since there is no that pesky DVI port to saw off.