3 ms·
> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 secon
by sho 26d ago
> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
- simonw 26d agoThe difference is memory bandwidth. The M5 Ultra that's coming out on 22nd September can do 1,200GB/s. The M5 Max you can buy today only has 614GB/s.
- sho 26d agoSo, that gets us to about where nVidia was with Ampere in 2020. Let's hope the M7 catches us up with at least Hopper.
- Gareth321 25d agoWhile true, the news here is the size of the unified RAM. Nvidia only exceeded 256GB RAM in the 2025 B300 - 288GB. The B300 alone (without the baseboard/PSU/chassis/wiring/CPUs/system RAM/etc) is at least 700% more expensive. This enables large language models on consumer hardware. 1200GB/s is plenty for many tasks.
- hypfer 25d ago> 1200GB/s is plenty for many tasks. This + due to the hardware being so prohibitively expensive, we're seeing software optimizations happening. Like that dflash2 stuff for example, or an LRU for MoE and all that kind of stuff.
- klooney 25d agoI wonder if the inference acceleration companies will ever produce a consumer product
- nojs 25d agoThe complaint is about prefill which is not memory bandwidth bound, it's compute bound. But they added neural accelerators for matmuls to the shader cores which should make prefill faster.
- chisleu 25d agoIt is and it isn't. Why are you comparing the m5max instead of the m4ultra? The big deal to me is the number of compute cores for prefill tps, which is suppose to be 4x faster on the m5ultra. It's my opinion that the m5 ultra is going to be a really big deal in terms of local AI accessibility. Flash sized models (~200-300b params) are going to be reasonably fast as long as you aren't throwing 40k context at it on each or the first request (ie, agentic harnesses). Even agentic harnesses like Cline should move at a reasonable clip on m5 ultra. I suppose we will know sooner than later. FYSA: Former m4 ultra 512GB owner and current 4x rtx6000 owner here. I upgraded because I needed more prompt processing speed and concurrency.
- UltraSane 25d agoWhat do you use all that local tokens/second for?
- big-and-small 25d agoI guess you mean M3 ultra since M4 ultra never existed.
- akmarinov 26d agoThe author put in the numbers, but maybe you didn’t read them. 45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio. Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash
- sho 25d agoThe joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous. There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
- EagnaIonat 25d ago> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds. Your knowledge is out of date. In truth it depends on the Mac and the models used. I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled. Question: "Give me a python code snippet that opens a file and sorts the lines of text. " In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each. Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
- desterothx 25d agoWe are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter
- EagnaIonat 25d agoI was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second. Local models are good enough that it's not an issue. But keep changing the goalposts if it makes you happy.
- ErroneousBosh 25d agoAlso, LLMs never *write* code snippets, they just pirate them from somewhere else.
- bsoqk 25d agoSo, like humans? Code didn't just appear in my brain, I learnt it from reading it everywhere else.
- mcr70 25d agonice answer. LLMs do reflect whatever humankind has produced. In good and in bad. But it is hardly pirating. It can also do what humankind has never failed to do, for example: https://www.nature.com/articles/d41586-026-02822-9 https://www.nature.com/articles/d41586-026-02822-9
- VCFundedGenYer 25d agoYou still wrote it, processed it in your brain, and reasoned about it. With an LLM you do zero of those things. I don't understand how you don't get that.
- ErroneousBosh 25d agoYes, but if you were being accurate you wouldn't say "AI wrote this code for me", you'd say "I used an Eliza bot to automate copying and pasting from someone else's github repo".
- zahlman 25d agoCan you show me an example of a time that you prompted an LLM to provide some code, it did so, and then you were able to track down an original source for the output?
- ErroneousBosh 25d agoEvery time I've asked an LLM to provide some code, it's given me an example from my own github repos. Granted, I've only asked it about some of the weirdly specific technological niche that my code inhabits.