3 ms·
A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations th
by ip26 11d ago
A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.
Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.
- nixon_why69 10d agoSpeculative decoding is software, though. How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
- searealist 10d ago2x is more typical, and only for dense models, not the sparse models all frontier labs use.