4 ms·
As far as benchmarketing goes they clearly went with prefill because it's much easier for apple to improve prefill numbers (flops-dominated) than decode (bandwi
by jbellis 7mo ago
As far as benchmarketing goes they clearly went with prefill because it's much easier for apple to improve prefill numbers (flops-dominated) than decode (bandwidth-dominated, at least for local inference); M5 unified memory bandwidth is only about 10% better than the M4.
- jtbaker 7mo agoOk, but prefill/prompt processing was definitely the weak point before. They were already solid in raw tokens/sec after TTFT