4 ms·
Prompt caching for cheaper LLM tokens
- Youden 10mo agoLink seems to be broken: content briefly loads then is replaced with "Something Went Wrong" then "D is not a function". Stays broken with adblock disabled.
- samwho 10mo agoAnother person had this problem as well and we couldn’t figure out what causes it. We suspect something to do with WebGL support. What browser/device are you using? Does it still break if you disable all extensions? I’d love to fix this.
- bkor 10mo agoIt gives "D is not a function". This on Firefox 146. Various extensions including Ublock Origin but that doesn't seem to cause it. Also doesn't work in a private window.
- est 10mo agoThis is a surprising good read of how LLM works in general.
- samwho 10mo agoIt’s funny, I didn’t set out for that to be the case. When I pitched the idea internally, I wanted to scratch my own itch (what on earth is a cached token?) and produce a good post. But then I realised I had to go deeper and deeper to get to my answer and accidentally made a very long explainer.
- yomismoaqui 10mo agoThanks for the post, it's near perfect in focus, detail and how it's written. EDIT: You have some minor typos in the post (psuedocode)
- aitchnyu 10mo agoTook me a minute to see it is same Ngrok which provided freemium tunnels to localhost. How did they adapt to the AI revolution?
- samwho 10mo agoIt is the same ngrok! The product has grown a lot since the mid 2010s. Still got free localhost tunnelling, but we also have a whole bunch of production-grade API gateway tooling and, as of recently, AI gateway stuff too.
- NooneAtAll3 10mo agoBlog starts loading and then gives "Something Went Wrong. D is not a function" error displayed
- belter 10mo agoYou should upgrade IE6. It has been out of support for a while...
- samwho 10mo agoCould you tell me what browser/OS/device you’re using? A few people have said this and I haven’t been able to reproduce it.
- NooneAtAll3 10mo agolibrewolf, fork of firefox, latest version f12 menu lists this: Loading failed for the <script> with source “https://global.ketchcdn.com/web/v2/config/ngrok/ngrok_ketch_tag/boot.js https://global.ketchcdn.com/web/v2/config/ngrok/ngrok_ketch_...”. prompt-caching:1:356 Response { status: 404, type: "default", url: "", redirected: false, ok: false, statusText: "Not Found", headers: Headers(1), body: ReadableStream, bodyUsed: false } React Router caught the following error during render entry.client-BTJ7ChVH.js:8:64676 Response { status: 404, type: "default", url: "", redirected: false, ok: false, statusText: "Not Found", headers: Headers(1), body: ReadableStream, bodyUsed: false } Uncaught Error: Minified React error #520; visit https://react.dev/errors/520 https://react.dev/errors/520 for the full message or use the non-minified dev environment for full errors and additional helpful warnings. chunk-G3INQAYP-D7BZozYw.js:4:2490 Rm https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:8 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... mu https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:8 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Lm https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:8 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... t1 https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:8 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... A1 https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:8 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Ba https://frontend-blog-ngrok.vercel.app/assets/entry.client-BTJ7ChVH.js:1 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Caused by: Response { … }
- tomhow 10mo ago[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 https://news.ycombinator.com/item?id=45988611 for explanation]
- simedw 10mo agoThanks for sharing; you clearly spent a lot of time making this easy to digest. I especially like the tokens-to-embedding visualisation. I recently had some trouble converting a HF transformer I trained with PyTorch to Core ML. I just couldn’t get the KV cache to work, which made it unusably slow after 50 tokens…
- samwho 10mo agoThank you so much <3 Yes, I recently wrote https://github.com/samwho/llmwalk https://github.com/samwho/llmwalk and had a similar experience with cache vs no cache. It’s so impactful.
- mrgaro 10mo agoHopefully you can write the teased next article about how Feedforward and Output layers work. The article was super helpful for me to get better understanding on how LLM GPTs work!
- samwho 10mo agoYeah! It’s planned for sure. It won’t be the direct next one, though. I’m taking a detour into another aspect of LLMs first. I’m really glad you liked it, and seriously the resources I link at the end are fantastic.
- coderintherye 10mo agoReally well done article. I'd note, when I gave the input/output screenshot to ChatGPT 5.2 it failed on it (with lots of colorful chain of thought), though Gemini got it right away.
- willvarfar 10mo agoA really clear explanation! So if I were running a provider I would be caching popular prefixes for questions across all users. There must be so many questions that start 'what is' or 'who was' etc? Also, can subsequences in the prompt be cached and reused? Or is it only prefixes? I mean, can you cache popular phrases that might appear in the middle of the prompt and reuse that somehow rather than needing to iterate through them token by token? E.g. must be lots of times that "and then tell me what" appears in the middle of a prompt?
- samwho 10mo agoWith KV caching as it’s described there it has to be a prefix match. OpenAI state in their docs they don’t cache anything below 1024 tokens long, and I’m sure I read somewhere that they only cache in 1024 token blocks (so 1024, 2048, 3072, etc) but I can’t find it now. There’s been some research into how to cache chunks in the middle, but I don’t think any of the providers are doing it yet because it needs the prompt to be structured in a very specific way.
- moebrowne 10mo agohttps://platform.openai.com/docs/guides/prompt-caching#requirements https://platform.openai.com/docs/guides/prompt-caching#requi... > Caching is available for prompts containing 1024 tokens or more. No mention of caching being in blocks of 1024 tokens thereafter.
- IanCal 10mo agoAt launch it was described as being in blocks of 128 https://openai.com/index/api-prompt-caching/ https://openai.com/index/api-prompt-caching/
- GeneralMayhem 10mo agoReally only prefixes, without a significant loss in accuracy. The point is that because later tokens can't influence earlier ones, the post-attention embeddings for those first tokens can't change. But the post-attention embeddings for "and then tell me what" would be wildly different for every prompt, because the embeddings for those tokens are affected by what came earlier. My favorite not-super-accurate mental model of what's going on with attention is that the model is sort of compressing the whole preceding context into each token. So the word "tell" would include a representation not just of the concept of telling, but also of what it is that's supposed to be told. That's explicitly what you don't want to cache. > So if I were running a provider I would be caching popular prefixes for questions across all users Unless you're injecting user context before the question. You can have a pre baked cache with the base system prompt, but not beyond that. Imagine that the prompt always starts with "SYSTEM: You are ChatGPT, a helpful assistant. The time is 6:51 ET on December 19, 2025. The user's name is John Smith. USER: Hi, I was wondering..." You can't cache the "Hi, I was wondering" part because it comes after a high-entropy component (timestamp and user name).
- duggan 10mo agoIt was a real facepalm moment when I realised we were busting the cache on every request by including date time near the top of the main prompt. Even just moving it to the bottom helped move a lot of our usage into cache. Probably went from something like 30-50% cached tokens to 50-70%.
- Havoc 10mo agoDoes anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot
- samwho 10mo agoI was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.
- samwho 10mo agoThe only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.
- gunalx 10mo agoI habe come across turning on caching means the llm has a faint memory of what was in the cache, even to unrelated queries. If this is the case its fully unreasonable to share the cache, because of possibility of information leakage.
- samwho 10mo agoHow would information leak, though? There’s no difference in the probability distribution the model outputs when caching vs not caching.
- sroussey 10mo agothe probability distribution the model outputs is identical under identical conditions. A local model running alone on your machine will 100% always return the exact same thing and the internal state will be exactly the same and you can checkpoint or cache that to avoid rerunning to that point. But… conditions can be different, and batching requests tends to affect other items in flight. I believe Thinking Machines had an article about how to make a request deterministic again without performance going to complete crap. I tend to think of things this way (completely not what happens though): what if you were to cache based on a tensor as the key? To generate a reasonably sized key what is an acceptable loss of precision to retrieve the same cache knowing that there is inherent jitter in the numbers of the tensor? And then the ever so slight leak of information. But also multiplied since there are internal kv caches for tokens and blah blah blah.
- WillAdams 10mo agoWhen will Microsoft do this sort of thing? It's a pain having to tell Copilot "Open in pages mode" each time it's launched, and then after processing a batch of files run into: https://old.reddit.com/r/Copilot/comments/1po2cuf/daily_limit_reached_how_to_keep_working_today/ https://old.reddit.com/r/Copilot/comments/1po2cuf/daily_limi...
- holbrad 10mo agoI gave the table of inputs and outputs to both Gemini 3.0 flash and GPT 5.2 instant and they were stumped. https://t3.chat/share/j2tnfwwful https://t3.chat/share/j2tnfwwful https://t3.chat/share/k1xhgisrw1 https://t3.chat/share/k1xhgisrw1
- dangoodmanUT 10mo agoBut why is this posted on ngrok?
- toobulkeh 10mo agoThey have an AI router they just released. ngrok.ai
- who-shot-jr 10mo agoWhat a fantastic article! How did you create the animations?
- simonw 10mo agoSam has a long history of building beautiful visual explanations like this - I didn't realize he works for ngrok now, here's his previous independent collection: https://samwho.dev/ https://samwho.dev/
- samwho 10mo agoSimon, you’re too kind. Thank you. <3
- samwho 10mo agoThank you! <3 These are all built with React and CSS animations (or the Web Animations API where I needed it). I’m not very good at React so the code is a real mess. 2 of the components also use threejs for the 3D bits. For the stuff on my personal site, which simonw graciously linked to in another reply, you can see all the code behind my work at https://github.com/samwho/visualisations https://github.com/samwho/visualisations
- deleted 10mo ago[deleted]