2 ms·
OpenAI haven't documented if they do this, but it should be reasonably easy to determine via experiments against their API. Claude's prompt caching gives a ver
by simonw 2y ago
OpenAI haven't documented if they do this, but it should be reasonably easy to determine via experiments against their API.
Claude's prompt caching gives a very material performance improvement - they claim up to a 4x performance boost in https://www.anthropic.com/news/prompt-caching https://www.anthropic.com/news/prompt-caching - so if OpenAI have similar techniques, even undocumented, they should become visible through sending the same prompt a bunch of times and measuring the latency.
- NavinF 2y agoOpenAI docs say they also improve latency with caching: https://news.ycombinator.com/item?id=41302358 https://news.ycombinator.com/item?id=41302358