4 ms·
This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. I am not an exp
by password4321 4mo ago
This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. I am not an expert but it sounds like your "one simple trick" would fix context issues and allow much tighter control over token usage. Thanks for being willing to share this tip in an HN comment, changing how those in the know use AI agents going forward -- it's hard to keep up!
- Jgrubb 4mo agoThe tokens are still being burnt, they're just doing so in a parallel dimension from the users main context window.
- ViewTrick1002 4mo agoThe real benefit is being able to use a cheaper, but good enough, model with a specific system prompt dedicated to that task.
- ajmurmann 4mo agoIt's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
- knollimar 4mo agoDon't you resend after every turn, so splitting it avoids the n^2 token usage (granted it's cached so there's some optimal amount here)
- ajmurmann 4mo agoYes, exactly. You resend it on every turn (assuming no cache hits). This is why using the shorter-lived subagent to take in that context and only return the useful result back to the longer-lived context safes tokens.
- loeg 4mo ago> This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. AI vendors still need to compete with each other both in terms of token cost and competency. An agent that is costly and less effective by wasting tokens is less competitive.