4 ms·
full riser cables like they used doesn't impact performance. Hanging it off on open air frame IMO is better, keeps everything cooler, not just the GPU but the
by segmondy 3y ago
full riser cables like they used doesn't impact performance. Hanging it off on open air frame IMO is better, keeps everything cooler, not just the GPU but the motherboard and surrounding components. With only 2 24gb GPU they are not going to be able to run larger models. You can't experiment with 70b models without offloading to CPU which is super slow. The best models are 70b+ models.
- ImprobableTruth 3y ago48 gb suffice for 4-bit inference and q-lora training of a 70b model. ~80 GB allows you to push it to 8-bit (which is nice of course), but full precision finetuning is completely out of reach either way. Though you're right of course that pcie will totally suffice for this case.