2 ms·

Thanks! Is there something about their platform that prevents caching? Or are they just not passing on the discount?

• eli 1mo ago
  The session had a 91.4% cache hit rate. They just give zero discount.
• imtringued 1mo ago
  It sounds to me that they don't have enough capacity and they want to discourage people from using the service. There is nothing about their architecture that prevents reusing the KV cache other than the opportunity cost of keeping the memory occupied.