3 ms·
I thought OpenAI would still handle case? Their cache would work up to the end of the file and you would then pay for uncached tokens for the user's question. H
by simonw 1y ago
I thought OpenAI would still handle case? Their cache would work up to the end of the file and you would then pay for uncached tokens for the user's question. Have I misunderstood how their caching works?
- thefroh 1y agonot if call #1 is the file + the question, call #2 is the file + a different question, no. if call #1 is the file, call #2 is the file + the question, call #3 is the file + a different question, then yes. and consider that "the file" can equally be a lengthy chat history, especially after the cache TTL has elapsed.
- simonw 1y agoI vibe-coded up a quick UI for exploring this: https://tools.simonwillison.net/prompt-caching https://tools.simonwillison.net/prompt-caching As far as I can tell it will indeed reuse the cache up to the point, so this works: Prompt A + B + C - uncached Prompt A + B + D - uses cache for A + B Prompt A + E - uses cache for A