4 ms·

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your

by NBJack 2y ago

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1M?

config_yml 2y ago
I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.
- BonoboIO 2y ago
  100k … I hear and use all of these high token context LLMs, but often they fail to include all information that should be in their context. Does your approach work for you?
- social_quotient 2y ago
  Would love if you could de identify and share a bit more detail on a prompt that big.
minimaxir 2y ago
The documentation is here and has better examples: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-... tl;dr can cache system prompts, tools, and user messages (up to 4 total) with better returns on massive inputs such as documents. The use case is more for client-facing applications that would hit the cache frequently rather than internal copilots.
calendarsnack 2y ago
Here is a simple use-case that comes to mind. Let's say you have a medium-size repository, such that all the source files can fit in the context. You want to use Claude as an advanced auto-complete in a certain file, but it's important that it has visibility to other files in the repo. You can put all the other source files in an initial system message and the current file in the user message. Then, if you call multiple autocompletes within 5 minutes of each other, you pay a drastically reduced price for including all of the other files in the context. Also, the latency is much reduced! Yes, you could probably get a similar outcome by incorporating RAG, search tools etc to the autocomplete, but as a simple approach with fewer moving parts, the caching will reduce costs for this setup.
deleted 2y ago
[deleted]
Tiberium 2y ago
>it assumes the exact same context window across requests That is not true, caching works across multiple requests, that's why it's so good. You can do 5 different concurrent requests and they'll all get cached and cache read if the cache is still warm for them.
Sakos 2y ago
It says the timeout is refreshed if there's a cache hit within the 5 minutes, and the base cost is 10% of what it would be otherwise. Seems pretty damn useful to me. What seems useless to you exactly? I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation). Simple queries are where I expect there to be the least gain for this kind of thing.
knallfrosch 2y ago
Think beyond AI coding assistants. JSON schema definitions, FAQs, product manuals, game instructions, game state, querying a student's thesis.. anything where users query a chatbot information about something specific.
simonw 2y ago
It’s useful to consider the impact this has on conversations. In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time. Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.

by NBJack 2y ago

config_yml 2y ago

I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.

BonoboIO 2y ago
100k … I hear and use all of these high token context LLMs, but often they fail to include all information that should be in their context. Does your approach work for you?
social_quotient 2y ago
Would love if you could de identify and share a bit more detail on a prompt that big.

minimaxir 2y ago

The documentation is here and has better examples: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-... tl;dr can cache system prompts, tools, and user messages (up to 4 total) with better returns on massive inputs such as documents. The use case is more for client-facing applications that would hit the cache frequently rather than internal copilots.

calendarsnack 2y ago

Here is a simple use-case that comes to mind. Let's say you have a medium-size repository, such that all the source files can fit in the context. You want to use Claude as an advanced auto-complete in a certain file, but it's important that it has visibility to other files in the repo. You can put all the other source files in an initial system message and the current file in the user message. Then, if you call multiple autocompletes within 5 minutes of each other, you pay a drastically reduced price for including all of the other files in the context. Also, the latency is much reduced! Yes, you could probably get a similar outcome by incorporating RAG, search tools etc to the autocomplete, but as a simple approach with fewer moving parts, the caching will reduce costs for this setup.

deleted 2y ago

[deleted]

Tiberium 2y ago

>it assumes the exact same context window across requests That is not true, caching works across multiple requests, that's why it's so good. You can do 5 different concurrent requests and they'll all get cached and cache read if the cache is still warm for them.

Sakos 2y ago

It says the timeout is refreshed if there's a cache hit within the 5 minutes, and the base cost is 10% of what it would be otherwise. Seems pretty damn useful to me. What seems useless to you exactly? I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation). Simple queries are where I expect there to be the least gain for this kind of thing.

knallfrosch 2y ago

Think beyond AI coding assistants. JSON schema definitions, FAQs, product manuals, game instructions, game state, querying a student's thesis.. anything where users query a chatbot information about something specific.

simonw 2y ago

It’s useful to consider the impact this has on conversations. In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time. Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.