3 ms·
Despite the hype, the 512GB Mac is not really a good buy for LLMs. The ability to run a giant model on it is a novelty that will wear off quickly... it's just t
by Gracana 2y ago
Despite the hype, the 512GB Mac is not really a good buy for LLMs. The ability to run a giant model on it is a novelty that will wear off quickly... it's just too slow to run them at that size, and in practice it has the same sweet spot of 30-70B that you'd have with a much cheaper machine with a GPU, without the advantage of being able to run smaller models at full-GPU-accelerated speed.
- SV_BubbleTime 2y agoThere’s so much flux in LLM requirements. 2 to 3 tokens per second was actually probably fine for most things last year. Now, with reasoning and deep searching, research models, you’re gonna generate 1000 or more tokens just as it’s talking to itself to figure out what to do for you. So everyone’s focused on how big a model you can fit inside your ram, the inference speed is now more important than it was.
- Gracana 2y agoAbsolutely. The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need it to run ten times faster. I guess code is tough too. If I'm talking to a model I'll read everything it says, so 10-20 tok/s is well and good, but that's molasses slow if it's outputting code and I'm scanning it to see if it looks right.
- adgjlsfhk1 2y agocounterpoint: thinking models are good since they give similar quality at smaller RAM sizes. if a 16b thinking model is as good as a 60b one shot model, you can use more compute without as much RAM bottleneck
- terribleperson 2y agoCounter-counterpoint: RAM costs are coming down fast this year. Compute, not so much. I still agree, though.
- aurareturn 2y agoIt runs DeepSeek R1 q4 MoE well enough.
- Gracana 2y agoIt does have an edge on being able to run large MoE models.