3 ms·
Thats only the case with KV Cache and we do not know how and how long providers keep it.
by Fazebooking 9mo ago
Thats only the case with KV Cache and we do not know how and how long providers keep it.
- bluegatty 9mo agoPrefill is 10x faster than generation without caching, and 100x faster with caching - as a very crude measure. So it's not a matter of 'only the case'. Those are different scenarios. Some hosts are better than others with respect to managing caching, but the better one's provide decent SLA on that.