3 ms·
You could set up checkpoints where you store off the current state, and roll back to those checkpoints?
by StilesCrisis 2mo ago
You could set up checkpoints where you store off the current state, and roll back to those checkpoints?
- Aissen 2mo agoDefinitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
- Aissen 2mo agoI'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - trigger CoT, save result as a summary - go back to previous checkpoint - instead of generating tokens, add result of CoT summary as input tokens - continue normally For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc. This idea is free. I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.