4 ms·
I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level co
by bob1029 5mo ago
I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller.
I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread.
There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time.
Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.
- Etheryte 5mo agoHow do you get the agent to stick to it without constantly rejecting tool calls with the same description? I've tried a similar setup a number of times and it tends to forget about this constraint very quickly.
- embedding-shape 5mo agoIf the model isn't following the system/developer prompts easily, you might want to try a bigger/better model, tends to mostly be about model quality if it doesn't follow what you tell it to. Besides that, conflicting directions in the system/developer prompts can lead to the model seemingly ignoring instructions too.
- bob1029 5mo agoThe tool itself enforces the constraint. This is deterministic. If an agent tries to read a big fat file in root, it gets an error from that tool's implementation that reiterates the requirement. I don't bother warning it in the system prompt anymore. It's pointless. I let it bump its head as required. A few hundred tokens and the agent is back on track each time.
- WithinReason 5mo agoWhich tools? Even file reads and writes?
- bob1029 5mo agoEspecially these things. The only tools permissible to root in my scheme are call() and return().
- WithinReason 5mo agoIs it in pi.dev? Don't thinking tokens still take up context?
- gbro3n 5mo agoThis makes intuitive sense. Can I ask what harness you're using that allows you to configure the constraint and how?
- deleted 5mo ago[deleted]
- bob1029 5mo agoIt's a custom agent loop. There are no other parties involved here. Just vanilla C#/.NET and the OpenAI DLL.
- hadlock 5mo agoAre you going to open source it
- el_benhameen 5mo agoI would also be really interested in seeing this if you’re willing to share it.
- verdverm 5mo agoYou can do this in opencode and pi (haven't used), by defining your own agents or overriding the built-in ones, so in your primary agent you can disable all tools and give it good instructions for how to delegate I imagine most harnesses should have a way to do this today, if they don't, get a new one. OpenCode i.e. is highly customizable, Claude and VS Code both support a ton as well including custom agents (though unclear if you can create custom top-level in claude-code) https://opencode.ai/docs/agents/ https://opencode.ai/docs/agents/ https://code.claude.com/docs/en/sub-agents https://code.claude.com/docs/en/sub-agents https://code.visualstudio.com/docs/agent-customization/custom-agents https://code.visualstudio.com/docs/agent-customization/custo...
- gbro3n 5mo agoThanks, those don't deterministically prevent the main loop from using tools thought, unless I'm wrong that's just prompting the main agent on when to use specialized sub agents
- password4321 5mo agoThis is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. I am not an expert but it sounds like your "one simple trick" would fix context issues and allow much tighter control over token usage. Thanks for being willing to share this tip in an HN comment, changing how those in the know use AI agents going forward -- it's hard to keep up!
- Jgrubb 5mo agoThe tokens are still being burnt, they're just doing so in a parallel dimension from the users main context window.
- ViewTrick1002 5mo agoThe real benefit is being able to use a cheaper, but good enough, model with a specific system prompt dedicated to that task.
- ajmurmann 5mo agoIt's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
- knollimar 5mo agoDon't you resend after every turn, so splitting it avoids the n^2 token usage (granted it's cached so there's some optimal amount here)
- ajmurmann 5mo agoYes, exactly. You resend it on every turn (assuming no cache hits). This is why using the shorter-lived subagent to take in that context and only return the useful result back to the longer-lived context safes tokens.
- loeg 5mo ago
- KronisLV 5mo agoFor anyone using Claude Code, ask it to do all the work in workflows (it has a tool for that), they released that feature together with Opus 4.8 and it also seems a bit better at doing long tasks as well. The main conversation just orchestrates the work at that point.
- wrs 5mo agoYou can also just ask it to do work in a subagent. It will write a plan and launch the subagent to do the actual code, keeping it out of the main context. In addition, you can co-author a plan for a biggish chunk of work, divided into stages, have it launch a subagent for phase 1 and check its work, then ESC-ESC to go back to just after you wrote the plan and have it do phase 2. Repeat until done. This keeps the overall goal in the main context for the review, but clears out previous reviews. Kind of like a workflow but with more control.
- KronisLV 5mo agoMy problem with regular sub-agents is that after around 2-4 hours the main agent stops working on a task and asks for user input no matter how I tell it to continue autonomously until the ~5-15 stage plan is done, when when it has a clear plan that's made with the plan mode and instructions to continue autonomously. It's happened multiple times where I give it a task before going to sleep and when I come back it's stuck halfway through on some stupid summary, where my only response needed is basically "Yeah, continue." even though I use Opus. Using workflows for the higher level planning helped with that and those annoying pauses no longer happen, perhaps due to the main conversation being much shorter and apparently not enough for the weights to nudge towards user confirmation.
- wrs 4mo agoWell, 2-4 hours of autonomous work is outside my comfort zone to start with. But have you tried Claude Code’s “auto mode”? I haven’t seen this premature stopping since it was introduced.
- Muromec 5mo agoI have a different way, but still trying to figure out how well it works. Instead of going into recursion, the agent is allowed to restart the thread by doing the summarize/debrief/reflect pass, writing key findings into persistent memory and rewriting the prompt whenever the context goes too large or it gets stuck. Recursion with TCO if you may. In a way it's a generalization of the spec-driver approach, but in addition to the the formal spec the carryover buffer lives in the memory.
- stogot 5mo agoKiro does this automatically from what I can tell using it
- throwaway314155 5mo agoHow do you get something like this set up?
- nijave 5mo agoClaude Code seems to automatically do this in some cases. It seems to have some heuristic "will eat a lot of context" where it decides to dispatch a sub agent. I see it pretty frequently in troubleshooting and data analysis flows where it will dump the data collection and aggregation into a sub agent then pull out a summarized result. I'll do something similar where I have the main agent maintain context in a design doc/markdown file and update as it goes along. Then I can clear/restart/handoff at will
- port11 5mo agoMight depend on the model. Haiku doesn’t like to delegate unless you ask it to. I have a custom command for “delegate plan, delegate code, delegate review”, but launching it with Haiku gives me mediocre results.
- andai 5mo agoSo what does the top level thread look like? "Make foo() do bar" (Subagent invoked) "Job finished!"
- bob1029 5mo agoThe top level and N+1 looks like: [User] Actual human prompt [Agent] Attempted use of tool & hand slap [Agent] call(projection of user's prompt relative to discovered tool constraints) ["User"] Prompt from above call [Agent] Legal tool use [Agent] ... until satisfied [Agent] return(summary that satisfies the prompt for this level of execution) [Agent] Additional call() invokes possible depending on returned summary [Agent] Final return(summary) from root ends this turn of conversation and user sees summary [User] Next turn of conversation initiated by actual human
- kevincox 5mo ago> There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.
- pavlus 5mo agoI've actually tried something like that, with two tools: * push("What I'm about to do"), * pop("What I've achieved"). "Push" marks the position in current context after the call and returns "Proceed", "pop" erases everything after the matching call and replaces "Proceed" in it's result as what was passed to the argument to the "pop", effectively pruning long-winded head even inside one reasoning stream. In the end the model only sees how it decided to do work on something, and that it was already done, forgetting everything it between, except what it itself decided was important. Gemma 4 31B QAT successfully uses it when navigating a maze, marking positions at intersections, exploring them, navigating back and pushing again if necessary. Smaller models often fail to mark positions and forget to backtrack as well, instead they try to rely on themselves to track their paths and navigate back (and also fail). I think it should work for long-running deep research tasks, but I was too lazy to test it, because it all required a lot of code to glue this up, since most tools and libs are not designed to work like that, and now I'll need even more code to test it, without a purposeful task.