4 ms·
Author tries to avoid a database for storing tokens while the client is disconnected and ends up storing them in a pub/sub provider. There's no solution other
by nixpulvis 10mo ago
Author tries to avoid a database for storing tokens while the client is disconnected and ends up storing them in a pub/sub provider.
There's no solution other than to store the tokens somewhere, or drop them. You have to make a choice how long you want to allow reconnects for. And this is all pretty independent of the transport layer, as the author even mentioned themselves, you can resume even a new session as long as you have a prompt ID or something to tie it back to the original request.
I don't know enough about how the LLM providers stream results, but the original claim that inference is more expensive than transport is a good point, and caching tokens seems like a smart move. Unfortunately, we pay by the token, so I don't see the incentive for providers to spend time and money doing this for us.
- embedding-shape 10mo ago> Unfortunately, we pay by the token, so I don't see the incentive for providers to spend time and money doing this for us. Providing a better service, for one. Plenty of providers do offer caching, both input and output tokens, and usually give you a cheaper price for it too. Example from two of them: https://platform.claude.com/docs/en/build-with-claude/prompt-caching https://platform.claude.com/docs/en/build-with-claude/prompt... & https://api-docs.deepseek.com/guides/kv_cache https://api-docs.deepseek.com/guides/kv_cache
- nixpulvis 10mo agoI feel like it's slightly different to cache duplicate parts of the input, vs storing outputs when a connection drops.
- FuckButtons 10mo agoIt seems like a good use case for a caching layer. It seems like you would probably be able to make a set up for agentic systems more simply / cheaply in Hetzner than trying to cobble together a bunch of fragmented apis.