3 ms·

But memory bandwidth (bottleneck for LLM inference) is only marginally improved, 614 GB/s vs 546 GB/s for M4/M5 Max - where is this 4x improvement coming from?

by ivankra 7mo ago

But memory bandwidth (bottleneck for LLM inference) is only marginally improved, 614 GB/s vs 546 GB/s for M4/M5 Max - where is this 4x improvement coming from? I think I'll pass on upgrading.

singhrac 7mo ago
It’s prompt processing so prefill - that’s compute bound not memory.
0x457 7mo ago
4x is on Time To First Token it's on the graph.