4 ms·
Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better
by Aissen 1mo ago
Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
- Aissen 1mo agoI'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints : - use checkpoints to save KV cache before trigger CoT - trigger CoT, save result as a summary - go back to previous checkpoint - instead of generating tokens, add result of CoT summary as input tokens - continue normally For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc. This idea is free. I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.