4 ms·
in case people are unaware, this is more exciting than other random computer updates, since the Mac Studio is probably the best system in the world for running
by bananapub 2y ago
in case people are unaware, this is more exciting than other random computer updates, since the Mac Studio is probably the best system in the world for running LLMs, as it can come with >> 100GB of ram and that's all accessible to the graphics/neural accelerator at high speed.
this new one comes with up to 512GB of unified RAM!
- bryanlarsen 2y agoOr you could wait a couple months and get a Strix Halo based desktop from HP, Framework, Asus or GMK for > 100GB of unified memory.
- noelwelsh 2y agoDoesn't Strix Halo top out at 192GB RAM? The new M3 Ultra seems a lot more powerful on paper.
- Scramblejams 2y agoI believe it tops out at 128, of which 96 can be used as GPU VRAM. Hoping AMD will open the memory floodgates on the next rev.
- data-ottawa 2y ago110 with Linux, but 256gb/s memory bandwidth. The M3 Ultra seems strictly better but is also significantly more expensive.
- Scramblejams 2y ago110! Good to know, thanks for the info.
- kccqzy 2y agoI'm not aware that Strix Halo has shipped on desktop. I thought it was only shipped on laptops and tablets? And Framework announced a desktop but did not ship it yet.
- bryanlarsen 2y agoIt hasn't shipped anywhere yet. You can find previews of Strix Halo on laptops and tablets, but you can't buy them yet.
- short_sells_poo 2y agoIt has terrible compute performance...
- blobbers 2y agoBetter than cloud API calls?
- jazzyjackson 2y agoHave you used a 100GB model on a Mac studio? Tokens per second is single digit, I didn't find it usable at all, found myself going back to cloud APIs where 3000$ goes a much longer way I'm looking forward to trying Nvidia's little set top box if it actually ships, should have higher memory bandwidth, but still Ill probably set up a system where I email a query with attachments and just let DeepSeek email me back once it's finished with reasoning at 10T/s
- clonky 2y agoIt might blow your mind that you can run a quantized DeepSeek-R1 (671B) at over 15 t/s on an M2 Ultra 192GB and still get around 9000 context.
- angoragoats 2y agoI disagree with the “best system in the world for running LLMs” claim. The Mac platform offers high memory bandwidth, but especially with large models you start to quickly run into the fact that the CPU/GPU themselves are very slow compared to discrete GPUs. For models up to 70b parameters, I’d much rather use any PC with a couple of 3090/4090/5090s, and for anything bigger I’d rather 1) use cloud services and pay by the hour, or 2) run a larger Epyc/Xeon system with more GPUs if my use case absolutely requires local/offline support.