3 ms·
Or just mean that you could run a 5x bigger model on Apple than before.
by NetMageSCW 7mo ago
Or just mean that you could run a 5x bigger model on Apple than before.
- dragonwriter 7mo agoWell, since its kv-cache that TurboQuant optimizes, it means five times bigger context fits into RAM, all other things being equal, not a five times bigger model. But, sure, with any given context size and the same RAM available, you can instead fit a bigger model—which also takes more compute to get the same performance. Anything that increases the necessary compute to fully utilize RAM bandwidth in optimal LLM serving weakens Apples advantage for that.