4 ms·
Upshot of the paper -- right now KV caches are implemented for multiple deep layers in LLMs. Why not just the top layer? It would save memory. Initial result -
by vessenes 2y ago
Upshot of the paper -- right now KV caches are implemented for multiple deep layers in LLMs. Why not just the top layer? It would save memory.
Initial result -- those KV caches in lower layers matter, and output suffered.
Updated plan -- cull half the KV layers! This works 'nearly' as well as keeping all of them, with memory and compute savings.
Downside - triple the training, worse out of band / long context performance.
This feels to me like a technique you'd use on a particular architecture deployed at the edge where compute matters and you have a little extra room on performance. Phi-3 on raspberry pi, basically.
Interesting! As always, I wish models showed prompt output in their papers, not just perplexity numbers. But, here we are.
- adtac 2y ago> triple the training From what I understand, not quite. It looks like the cost of training might be similar, but less parallelisable within a specific token sequence. This is because they have to compute the KV of token T before they can use it in T+1 whereas in a regular training process you can compute the KV at each layer for every subsequence. You're right that it took 2.7x longer to train the smallest model but I wouldn't be surprised if the GPU utilisation was proportionally lower too.
- algo_trader 2y agoIn general, are the per-layer KV caches all independent of each other? If we split a model layer-wise for inference across X GPUs, does the cache split as well?
- YetAnotherNick 2y agoI wish papers could be accepted with negative results. There's a lot of value in not repeating the same mistakes specially in a field like deep learning, which is not an exact science and mostly just driven by intuition.