3 ms·
You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.
by broodbucket 3mo ago
You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.
- leemoore 3mo agoif you are refreshing a cache of 120k tokens, you have to input the same 120k tokens. you can't bounce the cache with 1 byte. A proper cache hit requires your entire context that is cached. if you change 1 byte in it, everything after that byte is a cache write and no longer a cache hit
- broodbucket 3mo agoYes, you're paying the cache read, not the cache write, which is much more expensive. You can do the math for how many 5min refreshes you can be afk for until it starts costing instead of saving