4 ms·
What kind of motherboard can accommodate such a set up? There are not enough PCIe slots on many high end boards.
by krisdol 4y ago
What kind of motherboard can accommodate such a set up? There are not enough PCIe slots on many high end boards.
- ftufek 4y agoIt's not even just about the slots, it's about the PCIe lanes (which is something I never had to worry until now, though I built countless PCs in the past). We tried bunch of setups with Threadrippers and EPYC, at the end settled for the ROMED8-2T which is a monster motherboard.
- tinco 4y agoWe run 4x 2080s on threadripper systems. What sort of trouble did you run into? I thought threadripper has plenty of PCIe lanes. We didn't have any trouble but it could be I missed something, we had to get it working quick and I didn't do very much benchmarking.
- ftufek 4y agoThreadrippers are great and I had 4x Threadripper setup for the longest time, but they are a bit more expensive. The advantage of EPYC is that because it's so common, we can find used cheaper ones on ebay. They are a bit slower I believe, but we can deal with that by using Nvidia's DALI and decoding images on the GPU rather than CPU.
- tinco 4y agoOhh I hadn't thought of there being cheaper ones on ebay. That's a good tip, I'll check it out for our next upgrade. We're doing photogrammetry not machine learning, running some blackbox software that scaled best with clock speed so threadrippers were the most efficient option. I put them in 4U supermicro boxes with a noctua cooler with 2 9000RPM delta fans attached to it with rip ties.
- BonoboIO 4y ago9000 RMP Delta They must scream and push air like hell
- knicholes 4y agoI just built a rig with a Romed8-2T as well. I got pcie 3 16x risers and zip-tied them above the tower into a rack shelf above it. It's super ghetto, and I can't believe it works, but it totally does. I'm hosting on vast.ai hoping someone will train with my 4-6 3090s, but everyone wants their large language models and image generation models that require more than the 24GB of RAM. shrug Maybe some day I'll use it myself to train my plastic surgery outcome estimation visualization GAN or diffusion model if I can figure out how to fine-tune one.
- galkk 4y agoDamn, could you share photo somewhere
- knicholes 4y agohttps://photos.app.goo.gl/tUg76BybihDSiLQH6 https://photos.app.goo.gl/tUg76BybihDSiLQH6
- johndough 4y agoI once had some zip ties which became brittle after a few months and would simply snap. Might be something to watch out for.
- ftufek 4y agoThere are very cheap high quality metal zip ties on Amazon, wouldn't recommend anything else since the heat from the card would just melt it.