4 ms·
You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.
by dannyw 3mo ago
You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.
- gruez 3mo agoRight, but also using the cache refreshes/extends it, which means you can pay for a write once, then have it extended for very cheap (1 token).
- spacemanspiff01 3mo agoYou dont have to repay per hour? It seems like that is a design issue - Like yes we will keep these in whatever cache you want, but you will have some sort of renewal period