2 ms·
OpenAI has a mysterious reference to caching in their guide to latency. > Maximize shared prompt prefix, by putting dynamic portions (e.g. RAG results, history
by KTibow 2y ago
OpenAI has a mysterious reference to caching in their guide to latency.
> Maximize shared prompt prefix, by putting dynamic portions (e.g. RAG results, history, etc) later in the prompt. This makes your request more KV cache-friendly (which most LLM providers use) and means fewer input tokens are processed on each request.
source: https://platform.openai.com/docs/guides/latency-optimization https://platform.openai.com/docs/guides/latency-optimization
- okl 2y agoSounds very similar to how e.g. docker/layers work - the later you put the more dynamic stuff, the higher the change that previous layers are reused. Meaning the cache entries are likely chained, i.e., each one depends on the preceding entry.