3 ms·
Do LLMs cache results now? I assume a lot of the same questions get asked, although the answer could depend on previous conversational context.
by mysterydip 1y ago
Do LLMs cache results now? I assume a lot of the same questions get asked, although the answer could depend on previous conversational context.
- make3 1y agomaybe you can do something like speculative decoding where you decode with a smaller model until the large model disagrees too much at checkpoints, but use the context free cache in place of a smaller LLM from the original method. you could also like do it multi level, fixed context free cache, small model, large model
- ethbr1 1y agoSomething like higher-dimensional Huffman compression for queries?
- cj 1y agoI imagine caching is directly in conflict with their desire to personalize chats by user. See: ChatGPT's memory features. Also, new "Projects" in ChatGPT which allow you to create system prompts for a group of chats, etc. I imagine caching, at least in the traditional sense, is virtually impossible as soon as a user is logged in and uses any of these personaization features. Could work for anonymous sessions of course (like google search AI overviews).