3 ms·
Sharding a dense model using Tensor Parallelism (TP) across dual RTX 5090s has a significantly worse performance penalty over PCIe than sharding an MoE model.
by seanmcdirmid 2mo ago
Sharding a dense model using Tensor Parallelism (TP) across dual RTX 5090s has a significantly worse performance penalty over PCIe than sharding an MoE model.
If you are using multiple GPUs, MoE is basically going to be your only workable choice unless you can leverage pipeline parallelism (only half your GPUs can work on a prompt at a time, so you need to process prompts back to back in a pipeline setup, and they better be doing similar things because your vram is limited).
- angoragoats 2mo agoHave you ever actually set up a multi-GPU system for inference? Based on my experience you are drastically overstating the problem. Both tensor and pipeline parallelism (without NVLink) produce a machine which is faster than any Mac on the planet, which is what we’re discussing here. Yes, each has pros and cons, and neither scales perfectly linearly. But it works great regardless.
- seanmcdirmid 2mo agoNo, and at $4000+ per 5090, I'm unlikely to have any experience anytime soon.
- angoragoats 2mo agoYou can parallelize inference with much cheaper GPUs as well! I’ve got a 4060 ti 16gb, and I’m thinking about getting another. I previously specced out a cluster using multiple 3090s. At the time, the 3090s were going for $700 on eBay. They’re more than that now, but there’s no need to spend $4k per GPU at all.