3 ms·
It is done at immediately before the LLM call, transforming the message history for the API call. This does reduce the context cache hit rate a bit, but I'm ca
by CuriouslyC 1y ago
It is done at immediately before the LLM call, transforming the message history for the API call.
This does reduce the context cache hit rate a bit, but I'm cache aware so I try to avoid repacking the early parts if I can help it. The tradeoff is 100% worth it though.
- psadri 1y agoI’m curious about this project (I’m working on something similar). Anyway to get in contact with you?
- CuriouslyC 1y agoyou can click my spam protected email links on https://sibylline.dev https://sibylline.dev, those should be working now. Any CTA will get me.