2 ms·

RAM price don't change anything. You can't fit an infinity of tokens in the KV cache, but the ones that are in there when a user request them are still practica

by stymaar 4mo ago

RAM price don't change anything. You can't fit an infinity of tokens in the KV cache, but the ones that are in there when a user request them are still practically free.

jgalt212 4mo ago
so the utility of the KV cache has nothing to do with available RAM?
- stymaar 4mo ago
  This proposition has no logical link with any of the above. Read again.