3 ms·
I think having a (small desktop) system with Strix Halo plus a GPU to accelerate prompt processing could be a good combo, avoiding the weakness of the Mac Ultra
by Tepix 2y ago
I think having a (small desktop) system with Strix Halo plus a GPU to accelerate prompt processing could be a good combo, avoiding the weakness of the Mac Ultra. The Strix Halo has 16 PCIe lanes.
- aurareturn 2y agoMax RAM for Strix Halo is 128GB. It's not a competitor to the Mac Ultra which goes up to 512GB. You shouldn't need another GPU to do prompt processing for Strix Halo since the biggest model it can realistically run is a 70B model. Prompt processing isn't going to help much because it has a good enough GPU but its memory bandwidth is only 256GB/s (~210 GB/s effective).
- izacus 2y ago> Max RAM for Strix Halo is 128GB. It's not a competitor to the Mac Ultra which goes up to 512GB. What a... strange statement. How did you get to that conclusion?
- aurareturn 2y agoWhy do you think it's strange?
- izacus 2y agoThe original poster arrogantly and confidently proclaims that a device that costs like 2000$ isn't going to be able to compete against a 10000$ SKU of another device. I'm wondering how do you get to such a conclusion?
- aurareturn 2y agoI honestly don't understand what you're trying to say.
- deleted 2y ago[deleted]
- porphyra 2y agoThe $2000 strix halo with 128 GB might not compete with the $9000 Mac Studio with 512 GB but is a competitor to the $4000 Mac Studio with 96 GB. The slow memory bandwidth is a bummer, though.
- aurareturn 2y agobut is a competitor to the $4000 Mac Studio with 96 GB. The slow memory bandwidth is a bummer, though. Not really. The M4 Max has 2x the GPU power, 2.13x the bandwidth, faster CPU. $2000 M4 Pro Mini is more of a direct comparison. The Mini only has 64GB max ram but realistically, a 32B model is the biggest model you want to run with less than 300 GB/s bandwidth.
- Tepix 2y agoYou will be limited to a much smaller context size with half the RAM even if you're using a smaller model.
- Tepix 2y agoRunning something like Qwq 32b q4 with a ~50k context will use up those 128GB with the large KV cache.
- aurareturn 2y agoSo what makes you think Strix Halo with such a weak GPU and slow memory bandwidth can handle 50k context with a usable experience for a 32B model? Let's be realistic here. The compute, bandwidth, capacity (if 128GB) are completely imbalanced for Strix Halo. M4 Pro with 64GB is much more balanced.
- Tepix 2y agoYou're probably right. However with sparse models and MoE, 128GB may be useful.
- Gracana 2y agoDespite the hype, the 512GB Mac is not really a good buy for LLMs. The ability to run a giant model on it is a novelty that will wear off quickly... it's just too slow to run them at that size, and in practice it has the same sweet spot of 30-70B that you'd have with a much cheaper machine with a GPU, without the advantage of being able to run smaller models at full-GPU-accelerated speed.
- SV_BubbleTime 2y agoThere’s so much flux in LLM requirements. 2 to 3 tokens per second was actually probably fine for most things last year. Now, with reasoning and deep searching, research models, you’re gonna generate 1000 or more tokens just as it’s talking to itself to figure out what to do for you. So everyone’s focused on how big a model you can fit inside your ram, the inference speed is now more important than it was.
- Gracana 2y agoAbsolutely. The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need it to run ten times faster. I guess code is tough too. If I'm talking to a model I'll read everything it says, so 10-20 tok/s is well and good, but that's molasses slow if it's outputting code and I'm scanning it to see if it looks right.
- adgjlsfhk1 2y agocounterpoint: thinking models are good since they give similar quality at smaller RAM sizes. if a 16b thinking model is as good as a 60b one shot model, you can use more compute without as much RAM bottleneck
- terribleperson 2y agoCounter-counterpoint: RAM costs are coming down fast this year. Compute, not so much. I still agree, though.
- Tepix 2y agoOf course it's a competitor. Only a fraction of M3 Ultra sold will have 512GB RAM
- nrp 2y agoNote that none of the PCIe interfaces on Strix Halo are larger than x4. The intent is to allow multiple NVMe drives and a Wi-Fi card. We also used PCIe for 5Gbit Ethernet.
- Scramblejams 2y agoLove what you're doing, I'm in batch 4! Feedback: That 4x slot looks like it's closed on the end. Can we get an open-ended slot there instead so we can choose to install cards with longer interfaces? That's often a useful fallback.
- sunshowers 2y agoHi Nirav! Long time admirer. Gen 4 x4 or gen 5 x4? I saw that gen 5 x4 results in maybe a 3% decrease in 5090 performance compared to gen 5 x16.
- nrp 2y agoGen 4 x4. That’s a limitation of Ryzen AI Max.
- sunshowers 2y agoThank you! Even gen 4 x4 doesn't look too bad, just a 6% drop at 4k with a 5090. I realize pairing this with a 5090 is a bit ridiculous, but... is it? https://www.techpowerup.com/review/nvidia-geforce-rtx-5090-pci-express-scaling/33.html https://www.techpowerup.com/review/nvidia-geforce-rtx-5090-p...
- Tepix 2y agoThat really depends on the use case, does it not?