3 ms·
It sounds to me that they don't have enough capacity and they want to discourage people from using the service. There is nothing about their architecture that
by imtringued 1mo ago
It sounds to me that they don't have enough capacity and they want to discourage people from using the service.
There is nothing about their architecture that prevents reusing the KV cache other than the opportunity cost of keeping the memory occupied.