4 ms·
What is stored is not the tokens, but all keys and values of all attention layers for each token.
by GaggiX 2y ago
What is stored is not the tokens, but all keys and values of all attention layers for each token.
- sdrg822 2y ago+1 it wouldn’t be terribly useful if it were only caching the tokenizer output.
- bastawhiz 2y agoAs I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.
- GaggiX 2y agoIt's probably much more than a single GB of data for a million tokens.
- hansonw 2y agoIt’s probably more. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1M 16-bit floats per token = 2MB. They have likely needed to implement some kind of KV compression (like DeepSeek) to make this even feasible.
- rfoo 2y agoIt's more likely some 400GB of data.
- karmasimida 2y agoThere are some errors in you calculation. > A token is 32-bit integer. No, in transformer, token is a vector, for larger models it is probably something like 6k-12k floats, assuming larger model sizes. Assume 8-bit precision, a token is more like 6-12kB, per token. So assume 100k tokens, you will end up with 554MB for input tokens, ALONE. Depending on your model architecture, the memory could vary, but from my observation, the runtime memory increase is at least on the same magnitude with the initial amount of memory usage upon loading the model, and this is for a moderate context length (<32k), and will grow linearly, if we don't count the n*n KV matrices. So you are easily looking at caching 10~100GB of data, in a very hot state, and that is going to be very expensive indeed.
- Kiro 2y agoIt's not. You're confused and don't understand what you're talking about. Just read all the replies you've already gotten and try to understand why you're wrong instead of doubling down on an incorrect take.