4 ms·
> ... re-ordered to take into account LLM memory patterns. If I understand you correctly, doesn't this break prefix KV caching?
by olejorgenb 1y ago
> ... re-ordered to take into account LLM memory patterns.
If I understand you correctly, doesn't this break prefix KV caching?
- CuriouslyC 1y agoIt is done at immediately before the LLM call, transforming the message history for the API call. This does reduce the context cache hit rate a bit, but I'm cache aware so I try to avoid repacking the early parts if I can help it. The tradeoff is 100% worth it though.
- psadri 1y agoI’m curious about this project (I’m working on something similar). Anyway to get in contact with you?
- CuriouslyC 1y agoyou can click my spam protected email links on https://sibylline.dev https://sibylline.dev, those should be working now. Any CTA will get me.