3 ms·
The "session is a document, not a log file" framing is the part I'd underline. Being able to inspect and edit the raw context JSON going to the model is crimina
by jabenhaim 3mo ago
The "session is a document, not a log file" framing is the part I'd underline. Being able to inspect and edit the raw context JSON going to the model is criminally rare, most tools treat the context as a black box, and on long runs context bloat is the single biggest cause of the model quietly drifting. Making it inspectable and prunable is a real lever, not a gimmick.
The thing I'm curious about: once you branch, backtrack, or edit a node in the CRDT tree, how do you reconcile that with the model's linear context on the next turn? If you reconstruct the transcript from the tree each turn, does editing anything upstream blow away provider prompt caching, since the Anthropic/OpenAI caches key off an exact prefix match? That's the tension I keep hitting: tree-shaped editing is exactly what you want for control, but it fights the flat-prefix caching that keeps long sessions cheap and fast. Curious whether you just eat the re-cache cost or do something cleverer.
Either way, Miller columns over a doom-scroll is the right instinct. Nice work.
- julesrms 3mo agoYeah, if you edit then you do blow the cache. Where I've found editing to be the most useful is e.g. when I've had a long task that got a bit rambling and digressed, and then I go away and need to resume it later. Leaving it means the provider will have lost the cache for it, but rather than spending a lot of tokens to /compact or just to continue the entire thread, you can go through and delete all the useless bits, maybe just leaving enough content in there for the LLM to figure out how to continue, and then get back on track quite cheaply