2 ms·
Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before
by ec109685 3mo ago
Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it.
We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point.
For example you never want your system prompt to start with the time that the session started. Move that to the first user message if needed.
- macNchz 3mo agoCaching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.
- marginalia_nu 3mo agoYeah there's quite a lot of potential bugs that could have this shape. If I were to guess it could be a buffer in a buffer pool not being sized and zeroed correctly, allowing stale data to bleed between sessions.
- nok22kon 3mo agoor the cache retrieval function for a key retrieving the wrong entry
- supriyo-biswas 3mo agoThere could just also be a bug where the output tokens of session 1 were shared with session 2, due to a race condition or similar.
- Waterluvian 3mo agoThere is a massive incentive for optimization, so I expect they’re doing a ton of very clever tricks, all of which make this kind of bug more likely.
- estebarb 3mo agoHash functions necesarily have collisions. Also, it is perfectly possible to introduce bugs in the hash function (hash inputs, hash function itself) that allows cross account contamination.
- margalabargala 3mo agoHash functions necessarily have collisions, but it's perfectly possible to make the expected time between collisions greater than the human lifespan.