3 ms·
> Mac studio wins in memory capacity, price, perf/watt and value. [citation needed]. I have personally specced out and built an nvidia GPU-based machine which
by angoragoats 2mo ago
> Mac studio wins in memory capacity, price, perf/watt and value.
[citation needed]. I have personally specced out and built an nvidia GPU-based machine which after some optimization, handily beat the Mac Studio in terms of tokens/watt for LLM inference with most models. This was in the M2 Ultra era, and I haven't run the numbers for the later generations, but nvidia's cards have gotten faster just as Apple's CPUs/GPUs have, so I would guess that it's still possible to do.
> RTX 6000 wins in performance, if your model can fit into the VRAM.
"if your model can fit into the VRAM" can be true for the Mac as well.
> There are very obvious and clear advantages to a Mac Studio.
There are certain advantages for sure, depending on your use case. They may _seem_ to be obvious, but as evidenced above, I believe that many people overestimate the Mac's superiority on the metrics you cite when comparing a Mac vs. a dedicated GPU for LLM inference.
- seanmcdirmid 2mo ago> "if your model can fit into the VRAM" can be true for the Mac as well. It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. Even going with two 5090s, you now have to shard your model and that is a PITA. But it turns out that MoE is the solution both for running models on macs of limited computer power means (not as fast as GPUs), and on multiple GPUs that require sharding the model.
- angoragoats 2mo ago> It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. I bristle at general statements like this when it obviously depends on the specific Mac and GPU in question. But yes, comparing a maxed out M5 Ultra with an RTX 6000, the Mac has much more memory. > Even going with two 5090s, you now have to shard your model and that is a PITA. Every modern tool does this for you automatically. It is absolutely not a pain in the least (e.g. llama.cpp ships with pipeline parallelism enabled by default).
- seanmcdirmid 2mo agoSharding a dense model using Tensor Parallelism (TP) across dual RTX 5090s has a significantly worse performance penalty over PCIe than sharding an MoE model. If you are using multiple GPUs, MoE is basically going to be your only workable choice unless you can leverage pipeline parallelism (only half your GPUs can work on a prompt at a time, so you need to process prompts back to back in a pipeline setup, and they better be doing similar things because your vram is limited).
- angoragoats 2mo agoHave you ever actually set up a multi-GPU system for inference? Based on my experience you are drastically overstating the problem. Both tensor and pipeline parallelism (without NVLink) produce a machine which is faster than any Mac on the planet, which is what we’re discussing here. Yes, each has pros and cons, and neither scales perfectly linearly. But it works great regardless.
- seanmcdirmid 2mo agoNo, and at $4000+ per 5090, I'm unlikely to have any experience anytime soon.
- angoragoats 2mo agoYou can parallelize inference with much cheaper GPUs as well! I’ve got a 4060 ti 16gb, and I’m thinking about getting another. I previously specced out a cluster using multiple 3090s. At the time, the 3090s were going for $700 on eBay. They’re more than that now, but there’s no need to spend $4k per GPU at all.
- aurareturn 2mo ago[citation needed]. No need. You can infer the logic with this line I wrote: RTX 6000 wins in performance, if your model can fit into the VRAM. I'm not sure what the controversy is here.
- angoragoats 2mo agoYou made claims about performance per watt and other metrics which were completely unsubstantiated and aren't backed up by the line you quoted. That's what I was asking for citations about.
- aurareturn 2mo agoWhat other metrics?
- angoragoats 2mo agoDo I need to quote your own statement to you, verbatim, again? > Mac studio wins in memory capacity, price, perf/watt and value.
- aurareturn 2mo agoMac Studio winning in memory capacity is just a fact. That's why I'm confused.
- angoragoats 2mo agoYes, I agree. What about the other metrics, including performance per watt which I’ve now mentioned four times and you’ve ignored the previous three times?
- aurareturn 2mo agohttps://www.youtube.com/watch?v=nwIZ5VI3Eus https://www.youtube.com/watch?v=nwIZ5VI3Eus This is a good video to watch. Yes, I agree. Glad you agree. I was just confused why you were questioning it. The big biggest advantage for Apple Silicon is that you can get much more VRAM per $ over Nvidia cards. No controversy.