2 ms·
Cache definitely isn't free! We're in a global RAM shortage and KV caches sit around consuming RAM in the hope that there will be a hit. The gamble with cachin
by mike_hearn 8mo ago
Cache definitely isn't free! We're in a global RAM shortage and KV caches sit around consuming RAM in the hope that there will be a hit.
The gamble with caching is to hold a KV cache in the hope that the user will (a) submit a prompt that can use it and (b) that will get routed to the right server which (c) won't be so busy at the time it can't handle the request. KV caches aren't small so if you lose that bet you've lost money (basically, the opportunity cost of using that RAM for something else).
- otterley 8mo agoWhy do you believe that caches are held in RAM? They don’t need RAM performance, and disk is orders of magnitude cheaper.
- mike_hearn 8mo agoBecause OpenAI specifically say that: https://developers.openai.com/api/docs/guides/prompt-caching/ https://developers.openai.com/api/docs/guides/prompt-caching... > When using the in-memory policy, cached prefixes generally remain active for 5 to 10 minutes of inactivity, up to a maximum of one hour. In-memory cached prefixes are only held within volatile GPU memory. You can opt-in to storing the caches on local disk but it's not the default. I haven't done the calculations for why they do this, but given that disaggregated parallel prefill and RDMA can recompute the KV cache very fast, you'd need a huge amount of bandwidth from disk to beat it (and flash drives wear out!).
- himata4113 8mo agofree relative to gpu cost even at these costs