4 ms·
A bit strange to use time to first token instead of throughput. Latency to the first token is not like a web page where first paint already has useful things t
by tux3 7mo ago
A bit strange to use time to first token instead of throughput.
Latency to the first token is not like a web page where first paint already has useful things to show. The first token is "The ", and you'll be very happy it's there in 50ms instead of 200ms... but then what you really want to know is how quickly you'll get the rest of the sentence (throughput)
- GeekyBear 7mo agoIn previous generations, throughout was excellent for an integrated GPU, but the time to first token was lacking.
- danudey 7mo agoSo throughput was already good but TTFT was the metric that needed more improvement?
- freehorse 7mo agoLet’s say TTFT needed the most improvement. At some point, loading the model with enough context size may take tens of seconds in some macs.
- convenwis 7mo agoGood is relative but first token was clearly the biggest limitation.
- zamadatix 7mo agoTo add to the sibling "good is relative" it also depends what you're running, not just your relative tolerances of what good is. E.g. in a MoE the decode speedup means the speed of prompt processing delay is more noticeable for the same size model in RAM.
- brookst 7mo agoYeah TTFT was terrible. I don’t think it’s unreasonable to benchmark the most-improved metric.
- fragmede 7mo agoNo you don't. Not as a sticky mushy human with emotions watching tokens drip in. There's a lot of feeling and emotion not backed by hard facts and data going around, and most people would rather see something happening even if it takes longer overall. Hence spinner.gif, that doesn't actually remotely do a damned thing, but it gives users reassurance that they're waiting for something good. So human psychology makes time to first token an important metric to look at, although it's not the only one.
- MrDrMcCoy 7mo agoSome kinds of spinners serve as a coal-mine canary indicating if the app has gotten wedged. Not hugely useful, but also not entirely useless.
- case540 7mo agoI assume it’s time to first output token so it’s basically throughput. How fast can it output 8001 tokens
- jbellis 7mo agoAs far as benchmarketing goes they clearly went with prefill because it's much easier for apple to improve prefill numbers (flops-dominated) than decode (bandwidth-dominated, at least for local inference); M5 unified memory bandwidth is only about 10% better than the M4.
- jtbaker 7mo agoOk, but prefill/prompt processing was definitely the weak point before. They were already solid in raw tokens/sec after TTFT
- hedgehog 7mo agoNot strange, for the kind of applications models at that size are often used for the prefill is the main factor in responsiveness. Large prompt, small completion.