3 ms·
No? You still pay for cache reads... Actually this is actually MORE expensive than just redoing prefill if your context is below 100k.
by himata4113 3mo ago
No? You still pay for cache reads... Actually this is actually MORE expensive than just redoing prefill if your context is below 100k.