4 ms·
This problem was already solved 10 years ago - crypto mining motherboards, which have a large number of PCIe slots, a CPU socket, one memory slot, and not much
by dist-epoch 11mo ago
This problem was already solved 10 years ago - crypto mining motherboards, which have a large number of PCIe slots, a CPU socket, one memory slot, and not much else.
> Asus made a crypto-mining motherboard that supports up to 20 GPUs
https://www.theverge.com/2018/5/30/17408610/asus-crypto-mining-motherboard-gpus https://www.theverge.com/2018/5/30/17408610/asus-crypto-mini...
For LLMs you'll probably want a different setup, with some memory too, some m.2 storage.
- jsheard 11mo agoThose only gave each GPU a single PCIe lane though, since crypto mining barely needed to move any data around. If your application doesn't fit that mould then you'll need a much, much more expensive platform.
- dist-epoch 11mo agoAfter you load the weights into the GPU and keep the KV cache there too, you don't need any other significant traffic.
- numpad0 11mo agoEven in tensor parallel modes? I thought it could only work if you're fine stalling all but n GPU for n users at any given moments.
- skhameneh 11mo agoIn theory, it’s only sufficient for pipeline parallel due to limited lanes and interconnect bandwidth. Generally, scalability on consumer GPUs falls off between 4-8 GPUs for most. Those running more GPUs are typically using a higher quantity of smaller GPUs for cost effectiveness.
- zozbot234 11mo agoM.2 is mostly just a different form factor for PCIe anyway.