3 ms·

> as input tokens that are already in the KV cache are practically free for the provider, not at today's RAM prices.

• stymaar 4mo ago
  RAM price don't change anything. You can't fit an infinity of tokens in the KV cache, but the ones that are in there when a user request them are still practically free.
  • jgalt212 4mo ago
    so the utility of the KV cache has nothing to do with available RAM?
    • stymaar 4mo ago
      This proposition has no logical link with any of the above. Read again.