3 ms·
That checks out in principle, but given that P40 doesn't support NVLink, I wouldn't count too much on using six of them together in a performant manner. But ye
by omneity 3y ago
That checks out in principle, but given that P40 doesn't support NVLink, I wouldn't count too much on using six of them together in a performant manner.
But yeah the best option remains an MI300 if you can afford that.
- a_wild_dandan 3y agoYeah, my M2 MacBook has 96GB @400GB/s. For $4k or so, it feels like cheating. Does it beat 4x24GB NVIDIA cards? Absolutely not! It's slower and occasionally runs into CUDA-moat software issues. But the capability to daily drive Mixtral 8x7 locally, with great token speeds, is phenomenal.
- AnthonyMouse 3y agoNVLink is most needed for training. For inference a lot of the popular models can usefully be run on multiple GPUs without it: https://www.reddit.com/r/LocalLLaMA/comments/142rm0m/llamacpp_multi_gpu_support_has_been_merged/ https://www.reddit.com/r/LocalLLaMA/comments/142rm0m/llamacp...