3 ms·
Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slo
by cyanydeez 3mo ago
Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user.
So yeah, when they banned tgird party harnesses there was a technical and $ case to have.