12 ms·

Prompt Caching

173 points by fallinditch 2y ago

NBJack 2y ago
Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1M?
- config_yml 2y ago
  I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.
  - BonoboIO 2y ago
    100k … I hear and use all of these high token context LLMs, but often they fail to include all information that should be in their context. Does your approach work for you?
  - social_quotient 2y ago
    Would love if you could de identify and share a bit more detail on a prompt that big.
- minimaxir 2y ago
  The documentation is here and has better examples: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-... tl;dr can cache system prompts, tools, and user messages (up to 4 total) with better returns on massive inputs such as documents. The use case is more for client-facing applications that would hit the cache frequently rather than internal copilots.
- calendarsnack 2y ago
  Here is a simple use-case that comes to mind. Let's say you have a medium-size repository, such that all the source files can fit in the context. You want to use Claude as an advanced auto-complete in a certain file, but it's important that it has visibility to other files in the repo. You can put all the other source files in an initial system message and the current file in the user message. Then, if you call multiple autocompletes within 5 minutes of each other, you pay a drastically reduced price for including all of the other files in the context. Also, the latency is much reduced! Yes, you could probably get a similar outcome by incorporating RAG, search tools etc to the autocomplete, but as a simple approach with fewer moving parts, the caching will reduce costs for this setup.
- deleted 2y ago
  [deleted]
- Tiberium 2y ago
  >it assumes the exact same context window across requests That is not true, caching works across multiple requests, that's why it's so good. You can do 5 different concurrent requests and they'll all get cached and cache read if the cache is still warm for them.
- Sakos 2y ago
  It says the timeout is refreshed if there's a cache hit within the 5 minutes, and the base cost is 10% of what it would be otherwise. Seems pretty damn useful to me. What seems useless to you exactly? I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation). Simple queries are where I expect there to be the least gain for this kind of thing.
- knallfrosch 2y ago
  Think beyond AI coding assistants. JSON schema definitions, FAQs, product manuals, game instructions, game state, querying a student's thesis.. anything where users query a chatbot information about something specific.
- simonw 2y ago
  It’s useful to consider the impact this has on conversations. In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time. Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.
mathgeek 2y ago
Direct link to the feature referenced: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
deleted 2y ago
[deleted]
Scene_Cast2 2y ago
Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workload.
- deleted 2y ago
  [deleted]
- maged 2y ago
  Why not? It's caching the state of the model after the cached prefix, so that inference workload doesn't need to be run again.
- pclmulqdq 2y ago
  You actually can cache the "output" of a transformer on the prefix by caching what happens in the attention layer for that text string (specifically the "K" and "V" tensors). Since the attention layer is a big part of the compute cost of the transformer, this does cut down FLOPs dramatically.
  - Scene_Cast2 2y ago
    Oh interesting, didn't know. How does this work past the first transformer in the stack?
    - lonk11 2y ago
      My understanding is that the attention in all transformer layers is "causal" - that is the output of a transformer layer for token N depends only on tokens from 0 to N. This means that every attention layer can use previously calculated outputs for the same prompt prefix. So it only needs to calculate from scratch starting from the first unique token in the prompt sequence.
    - danielmarkbruce 2y ago
nprateem 2y ago
I just tried Claude the other day. What a breath of fresh air after fighting the dogshit that is OpenAI. Far less "in the realm of", "in today's fast-moving...", multifaceted, delve or other pretentious wank. There is still some though so they obviously used the same dataset that's overweight in academic papers. Still, I'm hopeful I can finally get it to write stuff that doesn't sound like AI garbage. Kind of weird there's no moderation API though. Will they just cut me off if my customers try to write about things they don't like?
- vellum 2y ago
  You can try AWS Bedrock or Openrouter if that happens. They both have the Claude API.
- bastawhiz 2y ago
  > Will they just cut me off if my customers try to write about things they don't like? The response you get back will have a refusal, which is pretty standard
verdverm 2y ago
FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h
- bastawhiz 2y ago
  That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but it's still an outrageously high markup. They could do their own caching, not tell you they're doing it, and keep the difference and make even more money)
  - IanCal 2y ago
    Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching" on a tape somewhere? Yes. So where on this scale does keeping it quick to get to gpu memory lie? > That's two million times more expensive than the storage cost of standard S3 ( You're not comparing to s3 at all.
    - bastawhiz 2y ago
      "RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.
      - IanCal 2y ago
        Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.
rglover 2y ago
This is great news. Using Claude to build a new SaaS [1] and this will likely save me quite a bit on API costs. [1] https://x.com/codewithparrot https://x.com/codewithparrot
MaximusLegroom 2y ago
I guess they got tired of losing customers to Deepseek. They introduced this feature a while ago and their prices were already miniscule given that they only have to compute 20B active parameters.
WiSaGaN 2y ago
This feature was first introduced by deepseek. And deepseek will just do it automatically for you. https://platform.deepseek.com/api-docs/news/news0802/ https://platform.deepseek.com/api-docs/news/news0802/
- nextworddev 2y ago
  Yep. All the LLM providers fast follow aka copy each other within months, though
w10-1 2y ago
Setting aside efficiency or accuracy, caching enhances the value of prompt engineering and thus increases the effective value of AI services (how the value is monetized or split is TBD). Comments suggest that caching the state of the network might also reduce processing. I wonder if it also permits better A/B-style testing by reducing the effect of cross-domain errors. If the AI service providers made it easy to provide feedback on post-cache responses, the providers could incorporate the quality-enhancement loop accelerating time to product-market fit (at the risk of increasing dependency and reducing ability to switch).
politelemon 2y ago
Will this be making its way to Bedrock?
- kevsim 2y ago
  How is the Anthropic stuff on Bedrock in general? We're using OpenAI stuff on Azure right now and it's frustrating how slowly stuff gets rolled out in our region(s).
deleted 2y ago
[deleted]
xihajun 2y ago
Prompt + Lora? Train an adapter?

173 points by fallinditch 2y ago

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1M?

config_yml 2y ago
I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.
- BonoboIO 2y ago
  100k … I hear and use all of these high token context LLMs, but often they fail to include all information that should be in their context. Does your approach work for you?
- social_quotient 2y ago
  Would love if you could de identify and share a bit more detail on a prompt that big.
minimaxir 2y ago
The documentation is here and has better examples: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-... tl;dr can cache system prompts, tools, and user messages (up to 4 total) with better returns on massive inputs such as documents. The use case is more for client-facing applications that would hit the cache frequently rather than internal copilots.
calendarsnack 2y ago
Here is a simple use-case that comes to mind. Let's say you have a medium-size repository, such that all the source files can fit in the context. You want to use Claude as an advanced auto-complete in a certain file, but it's important that it has visibility to other files in the repo. You can put all the other source files in an initial system message and the current file in the user message. Then, if you call multiple autocompletes within 5 minutes of each other, you pay a drastically reduced price for including all of the other files in the context. Also, the latency is much reduced! Yes, you could probably get a similar outcome by incorporating RAG, search tools etc to the autocomplete, but as a simple approach with fewer moving parts, the caching will reduce costs for this setup.
deleted 2y ago
[deleted]
Tiberium 2y ago
>it assumes the exact same context window across requests That is not true, caching works across multiple requests, that's why it's so good. You can do 5 different concurrent requests and they'll all get cached and cache read if the cache is still warm for them.
Sakos 2y ago
It says the timeout is refreshed if there's a cache hit within the 5 minutes, and the base cost is 10% of what it would be otherwise. Seems pretty damn useful to me. What seems useless to you exactly? I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation). Simple queries are where I expect there to be the least gain for this kind of thing.
knallfrosch 2y ago
Think beyond AI coding assistants. JSON schema definitions, FAQs, product manuals, game instructions, game state, querying a student's thesis.. anything where users query a chatbot information about something specific.
simonw 2y ago
It’s useful to consider the impact this has on conversations. In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time. Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.

mathgeek 2y ago

Direct link to the feature referenced: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://docs.anthropic.com/en/docs/build-with-claude/prompt-...

deleted 2y ago

[deleted]

Scene_Cast2 2y ago

Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workload.

deleted 2y ago
[deleted]
maged 2y ago
Why not? It's caching the state of the model after the cached prefix, so that inference workload doesn't need to be run again.
pclmulqdq 2y ago
You actually can cache the "output" of a transformer on the prefix by caching what happens in the attention layer for that text string (specifically the "K" and "V" tensors). Since the attention layer is a big part of the compute cost of the transformer, this does cut down FLOPs dramatically.
- Scene_Cast2 2y ago
  Oh interesting, didn't know. How does this work past the first transformer in the stack?
  - lonk11 2y ago
    My understanding is that the attention in all transformer layers is "causal" - that is the output of a transformer layer for token N depends only on tokens from 0 to N. This means that every attention layer can use previously calculated outputs for the same prompt prefix. So it only needs to calculate from scratch starting from the first unique token in the prompt sequence.
  - danielmarkbruce 2y ago

nprateem 2y ago

I just tried Claude the other day. What a breath of fresh air after fighting the dogshit that is OpenAI. Far less "in the realm of", "in today's fast-moving...", multifaceted, delve or other pretentious wank. There is still some though so they obviously used the same dataset that's overweight in academic papers. Still, I'm hopeful I can finally get it to write stuff that doesn't sound like AI garbage. Kind of weird there's no moderation API though. Will they just cut me off if my customers try to write about things they don't like?

vellum 2y ago
You can try AWS Bedrock or Openrouter if that happens. They both have the Claude API.
bastawhiz 2y ago
> Will they just cut me off if my customers try to write about things they don't like? The response you get back will have a refusal, which is pretty standard

verdverm 2y ago

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

bastawhiz 2y ago
That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but it's still an outrageously high markup. They could do their own caching, not tell you they're doing it, and keep the difference and make even more money)
- IanCal 2y ago
  Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching" on a tape somewhere? Yes. So where on this scale does keeping it quick to get to gpu memory lie? > That's two million times more expensive than the storage cost of standard S3 ( You're not comparing to s3 at all.
  - bastawhiz 2y ago
    "RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.
    - IanCal 2y ago
      Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.

rglover 2y ago

This is great news. Using Claude to build a new SaaS [1] and this will likely save me quite a bit on API costs. [1] https://x.com/codewithparrot https://x.com/codewithparrot

MaximusLegroom 2y ago

I guess they got tired of losing customers to Deepseek. They introduced this feature a while ago and their prices were already miniscule given that they only have to compute 20B active parameters.

WiSaGaN 2y ago

This feature was first introduced by deepseek. And deepseek will just do it automatically for you. https://platform.deepseek.com/api-docs/news/news0802/ https://platform.deepseek.com/api-docs/news/news0802/

nextworddev 2y ago
Yep. All the LLM providers fast follow aka copy each other within months, though

w10-1 2y ago

Setting aside efficiency or accuracy, caching enhances the value of prompt engineering and thus increases the effective value of AI services (how the value is monetized or split is TBD). Comments suggest that caching the state of the network might also reduce processing. I wonder if it also permits better A/B-style testing by reducing the effect of cross-domain errors. If the AI service providers made it easy to provide feedback on post-cache responses, the providers could incorporate the quality-enhancement loop accelerating time to product-market fit (at the risk of increasing dependency and reducing ability to switch).

politelemon 2y ago

Will this be making its way to Bedrock?

kevsim 2y ago
How is the Anthropic stuff on Bedrock in general? We're using OpenAI stuff on Azure right now and it's frustrating how slowly stuff gets rolled out in our region(s).

deleted 2y ago

[deleted]

xihajun 2y ago

Prompt + Lora? Train an adapter?