7 ms·
I chased down what the "4x faster at AI tasks" was measuring: > Testing conducted by Apple in January 2026 using preproduction 13-inch and 15-inch MacBook Air
by jbellis 7mo ago
I chased down what the "4x faster at AI tasks" was measuring:
> Testing conducted by Apple in January 2026 using preproduction 13-inch and 15-inch MacBook Air systems with Apple M5, 10-core CPU, 10-core GPU, 32GB of unified memory, and 4TB SSD, and production 13-inch and 15-inch MacBook Air systems with Apple M4, 10-core CPU, 10-core GPU, 32GB of unified memory, and 2TB SSD. Time to first token measured with an 8K-token prompt using a 14-billion parameter model with 4-bit quantization, and LM Studio 0.4.1 (Build 1). Performance tests are conducted using specific computer systems and reflect the approximate performance of MacBook Air.
- azinman2 7mo agoSeems very reasonable to me
- tux3 7mo agoA bit strange to use time to first token instead of throughput. Latency to the first token is not like a web page where first paint already has useful things to show. The first token is "The ", and you'll be very happy it's there in 50ms instead of 200ms... but then what you really want to know is how quickly you'll get the rest of the sentence (throughput)
- GeekyBear 7mo agoIn previous generations, throughout was excellent for an integrated GPU, but the time to first token was lacking.
- danudey 7mo agoSo throughput was already good but TTFT was the metric that needed more improvement?
- freehorse 7mo agoLet’s say TTFT needed the most improvement. At some point, loading the model with enough context size may take tens of seconds in some macs.
- convenwis 7mo agoGood is relative but first token was clearly the biggest limitation.
- zamadatix 7mo agoTo add to the sibling "good is relative" it also depends what you're running, not just your relative tolerances of what good is. E.g. in a MoE the decode speedup means the speed of prompt processing delay is more noticeable for the same size model in RAM.
- brookst 7mo agoYeah TTFT was terrible. I don’t think it’s unreasonable to benchmark the most-improved metric.
- fragmede 7mo agoNo you don't. Not as a sticky mushy human with emotions watching tokens drip in. There's a lot of feeling and emotion not backed by hard facts and data going around, and most people would rather see something happening even if it takes longer overall. Hence spinner.gif, that doesn't actually remotely do a damned thing, but it gives users reassurance that they're waiting for something good. So human psychology makes time to first token an important metric to look at, although it's not the only one.
- MrDrMcCoy 7mo agoSome kinds of spinners serve as a coal-mine canary indicating if the app has gotten wedged. Not hugely useful, but also not entirely useless.
- case540 7mo agoI assume it’s time to first output token so it’s basically throughput. How fast can it output 8001 tokens
- jbellis 7mo agoAs far as benchmarketing goes they clearly went with prefill because it's much easier for apple to improve prefill numbers (flops-dominated) than decode (bandwidth-dominated, at least for local inference); M5 unified memory bandwidth is only about 10% better than the M4.
- jtbaker 7mo agoOk, but prefill/prompt processing was definitely the weak point before. They were already solid in raw tokens/sec after TTFT
- hedgehog 7mo agoNot strange, for the kind of applications models at that size are often used for the prefill is the main factor in responsiveness. Large prompt, small completion.
- nabakin 7mo agoI would consider it reasonable if this was 4x TTFT and Throughput, but it seems like it's only for TTFT.
- lastdong 7mo ago14-billion parameter model with 4-bit quantization seems rather small
- simlevesque 7mo agoIt's not much for a frontier AI but it can be a very useful specialized LLM.
- giancarlostoro 7mo agoOn my 24GB RAM M4 Pro MBP some models run very quickly through LM Studio to Zed, I was able to ask it to write some code. Course my fan starts spinning off like the worlds ending, but its still impressive what I can do 100% locally. I can't imagine on a more serious setup like the Mac Studio.
- efxhoy 7mo agoHow is the output quality of the smaller models?
- elsombrero 7mo agonot good enough for coding anything more than simple scripts. generally, the less parameters, the less knowledge they have.
- kraig911 7mo agowhat model were you using?
- giancarlostoro 7mo agoWrote about it here: https://news.ycombinator.com/item?id=47191915 https://news.ycombinator.com/item?id=47191915
- jbellis 7mo agoYour limitation after prefill is memory bandwidth. A maxed out Studio has less than a single 3090 (really).
- whynotmaybe 7mo agoQuite interesting that it's now a selling point just like fps in Crysis was a long time ago.
- re-thc 7mo agoNext is the fps of an AI playing Crysis.
- patates 7mo agoNow that you mentioned it, these macs could theoretically also run crysis if it supported arm and such! They should add that to the marketing material :)
- butILoveLife 7mo ago>Time to first token measured with an 8K-token prompt using a 14-billion parameter model with 4-bit quantization Oh dear 14B and 4-bit quant? There are going to be a lot of embarrassed programmers who need to explain to their engineering managers why their Macbook can't reasonably run LLMs like they said it could. (This already happened at my fortune 20 company lol)
- knicholes 7mo agoI wonder if Apple has foresight into locally running LLMs becoming sufficiently useful.
- b112 7mo agoThey do! "You're holding it wrong*
- DiscourseFan 7mo agoIt won’t handle serious tasks but I have Gemma 3 installed on my M2 Mac and it is good for most of my needs—-esp data I don’t want a corporation getting its hands on.
- andai 7mo agoWhat kind of tasks are you using it for? I haven't really found any uses for small models.
- DiscourseFan 7mo agoNothing like coding, just like relatively basic stuff. Idk its hard to explain but I use AI so frequently for work that I have a sense for what it is capable of.
- andai 7mo agoWhich size Gemma are you using? I should clarify that by small I mean in the 3-8B range. I haven't tested the 14-30B ones, my experience is only about the smaller ones. In my experience, small models are not good for coding (except very basic tasks), they're not good for general knowledge. So the only purpose I could see for them would be, when they're given the information, i.e. summarization or RAG. But in my summarization experiments, they consistently misunderstood the information given to them. They constantly made basic errors and failed to understand the text. So having eliminated programming, general knowledge, summarization and (by extension, RAG, because if you can't understand the information, then you can't do RAG either, by definition) -- I have eliminated all the use cases that I had in mind! That would leave very basic tasks like classification or keywords, but I think there they would be in the awkward middle ground of being disappointing relative to big LLMs for many tasks, and cumbersome relative to small specialized models which can run fast and cheap and be fine tuned.
- fulafel 7mo agoSo it's not measuring output tokens/s, just how long it takes to start generating tokens. Seems we'll have to wait for independent benchmarks to get useful numbers.
- dotancohen 7mo agoFor many workflows involving real time human interaction, such as voice assistant, this is the most important metric. Very few tasks are as sensitive to quality, once a certain response quality threshold has been achieved, as is the software planning and writing tasks that most HN readers are likely familiar.
- raw_anon_1111 7mo agoThe way that voice assistants work even in the age of LLMs are: Voice —> Speech to Text -> LLM to determine intent -> JSON -> API call -> response -> LLM -> text to speech. TTFT is irrelevant, you have to process everything through the pipeline before you can generate a response. A fast model is more important than a good model Source: I do this kind of stuff for call centers. Yes I know modern LLMs don’t go through the voice -> text -> LLM -> text -> voice anymore. But that only works when you don’t have to call external sources
- garbageman 7mo agoI'm curious, what does the 'determine intent' mean in this case?
- raw_anon_1111 7mo agoAn “intent” is something that a person wants to do - set a timer, get directions, etc. A “slot” is the variable part of an intent. For instance “I want directions to 555 MockingBird Lane”. Would trigger a Directions intent that required where you are coming from and where you are going. Of course in that case it would assume your location. Back in the pre LLM days and the way that Siri still works, someone had to manually list all of the different “utterances” that should trigger the intent - “Take me to {x}”,”I want to go to {x}” in every supported language and then had to have follow up phrases if someone just said something like “I need directions” to ask them something like “Where are you trying to go”. Now you can do that with an LLM and some prompting and the LLM will keep going back and forth until all of the slots are filled and then tell it to create a JSON response when it has all of the information your API needs and you call your API. This us what a prompt would look like to use a book a flight tool. https://chatgpt.com/share/69a7d19f-494c-8010-8e9e-4e450f0bf05f https://chatgpt.com/share/69a7d19f-494c-8010-8e9e-4e450f0bf0... You also get the benefit of this works in any language not just English.
- gslepak 7mo agoThat is talking about battery life, not AI tasks. Footnote 53, where it says, "Up to 18 hours battery life": https://www.apple.com/macbook-pro/ https://www.apple.com/macbook-pro/
- Havoc 7mo agoDoes that include loading the model again? Apple seems to be the only company doing such shenanigans in their measurements
- easygenes 7mo agoTopical. My hobby project this week (0) has been hyper-optimizing microgpt for M5's CPU cores (and comparing to MLX performance). Wonder if anything changes under the regime I've been chasing with these new chips. 0: https://entrpi.github.io/eemicrogpt/ https://entrpi.github.io/eemicrogpt/
- gok 7mo agoconsider using fp16 or bf16 for the matrix math (in SME you can use svmopa_za16_f16_m or svmopa_za16_bf16_m)
- nullbyte808 7mo agoLike saying my PC boots up 2x faster so it must be 2x more powerful. lol
- Tostino 7mo agoIt was also one of the areas it was weakest in though, so this brings it way more in line with usable GPU territory.
- fnordpiglet 7mo agoI think the key stats are this (for m5 max) M5 128GB RAM with 614GB/s memory transfer This is a huge step over M4 32GB 153GB/s memory transfer For local LLM this make it a replacement for a DGX Spark, which offers a third of the transfer speed and is not something you toss in your backpack as your laptop. It’s practically useful for a lot of local use cases and that I think is the 4x factor (memory xfer) - but the 128Gb unified headroom tremendously improves the models you can run and training you can do.
- sroussey 7mo agoYou are comparing a M5 Max to a base M4. The M4 Max has 546 GB/s compared to 614GB/s for the M5 Max. Which is like 12% faster not 4x.
- zmmmmm 7mo agoWhat is truly amazing is the M1 Max is 400GB/s. 5 years later and we still only hit 1.5x on memory bandwidth. It's quite fascinating how high Apple spec'd it back then with apparently little foreknowledge of how important memory bandwidth would become, and then conversely how little they've managed to improve it now when it's so obvious how important it is.
- ColonelPhantom 7mo agoThe reason for that is that most memory bandwidth bumps come with new memory generations. For example an early DDR4 platform (e.g. Intel Skylake/Core iX-6000) and a late one (e.g. AMD Zen3/Ryzen 5000) only differ by 1.5x as well, typically. The same trend is visible in GPUs: for example, my RTX 2070 (GDDR6) has the same memory bandwidth as a 3070 and only a little bit less than a 4070 (GDDR6X). However, a 5070 does get significantly more bandwidth due to the jump to GDDR7. Lower-end cards like the 4060 even stuck to GDDR6, which gave them a bandwidth deficit compared to a 3060 due to the narrower memory buses on the 40 series.
- zmmmmm 7mo ago
- hydroreadsstuff 7mo agoThe 4x comes from the neural accelerators (tensor core in NVIDIA jargon). It's 4x fp16 over the vector path (And 8x compared to M1 because at some point they 2x'd the fp16 vector path). Therefore LLM prefill(context processing/TTFT), diffusion models (image gen), and e.g. video and photo effects that make use of them can be up to 4x faster. At fp16 that's the same speed at the same clock as NVIDIA. But NVIDIA still has 2xfp8 and 4xnvfp4. Batch-1 token generation, that is often quoted, does not benefit from this. It's purely RAM bandwidth-limited.