2 ms·
How do you know it's not prompt caching in the backend?
by htrp 2y ago
How do you know it's not prompt caching in the backend?
- simonw 2y agoOpenAI haven't documented if they do this, but it should be reasonably easy to determine via experiments against their API. Claude's prompt caching gives a very material performance improvement - they claim up to a 4x performance boost in https://www.anthropic.com/news/prompt-caching https://www.anthropic.com/news/prompt-caching - so if OpenAI have similar techniques, even undocumented, they should become visible through sending the same prompt a bunch of times and measuring the latency.
- NavinF 2y agoOpenAI docs say they also improve latency with caching: https://news.ycombinator.com/item?id=41302358 https://news.ycombinator.com/item?id=41302358
- nprateem 2y agoIMO they are definitely doing something. I've had countless occasions where the first few times I run a prompt I get excellent output, then several tries later it just returns the same dumb-sounding AI shite.