3 ms·
For large VRAM models, what about selling one of the 3090s, and putting the money towards an NVLink and a motherboard with two x16 PCIe slots (and preferably sp
by neilv 3y ago
For large VRAM models, what about selling one of the 3090s, and putting the money towards an NVLink and a motherboard with two x16 PCIe slots (and preferably spaced so you don't need riser cables)?
- nick7376182 3y ago[dead]
- p1esk 3y agoWhy do you need x16 pcie slots if you can use nvlink?
- elorant 3y agoNVlink is to connect the cards to each other. To connect them to the board you need the PCie slots.
- p1esk 3y agoWe are talking about increasing the intercard bandwidth, assuming that’s a bottleneck. It can be done by either increasing pcie bandwidth, or using nvlink. If you use nvlink, increasing pcie does not provide any additional benefit because nvlink is much faster than pcie. p.s. the mobo (B450 Steel Legend) already has 2 pcie x16 slots, so the recommendation does not make sense to me.
- segmondy 3y agofull riser cables like they used doesn't impact performance. Hanging it off on open air frame IMO is better, keeps everything cooler, not just the GPU but the motherboard and surrounding components. With only 2 24gb GPU they are not going to be able to run larger models. You can't experiment with 70b models without offloading to CPU which is super slow. The best models are 70b+ models.
- ImprobableTruth 3y ago48 gb suffice for 4-bit inference and q-lora training of a 70b model. ~80 GB allows you to push it to 8-bit (which is nice of course), but full precision finetuning is completely out of reach either way. Though you're right of course that pcie will totally suffice for this case.
- ImprobableTruth 3y agoIME NVLink would be overkill for this. Model parallelism means you only need bandwidth to transfer the intermediate activations (/gradients + optimizer state) at the seams and inference speed is generally slow enough that even pcie x8 won't be a bottleneck.