3 ms·

Yes, this is a core part of my workflow; use the smartest model (e.g. Fable) to generate a large hierarchical implementation plan, then clear the context and ha

Yes, this is a core part of my workflow; use the smartest model (e.g. Fable) to generate a large hierarchical implementation plan, then clear the context and have a lesser model (e.g. Sonnet) track and execute the plan [0], often in parallel. It all depends on how much work you're doing; if it's a simple task, there's virtually no need for plans at all; just have the smart agent do it. But if the work is going to take 5, 10 or 100 sessions, there's real value in the "smart agent plans, cheap agent executes" model. [0]: https://github.com/bensyverson/jobs https://github.com/bensyverson/jobs

ricardobeat 3mo ago
This is exactly what the post argues against though, as it leads to higher overall cost.
- bensyverson 3mo ago
  Yes, and I'm arguing that the post is over-simplifying. As I mention, for small plans, it's probably easier to have the big model just do it. But for large, multi-session plans, it's cheaper to let smaller models execute the bulk of the work.
  - markust 3mo ago
    Perhaps, there's a small change that might work better here - if we use a different agent (by creating custom agents) for plan and implementation, then the switch to the implementation agent wouldn't end up re-reading a lot as the new agent would have fresh empty context. I implemented this[1] from one of the posters on HN and I think it'd work well with this problem too. Would be great if you could point out if my understanding is wrong. [1]: https://www.stavros.io/posts/how-i-write-software-with-llms/ https://www.stavros.io/posts/how-i-write-software-with-llms/
    - bensyverson 3mo ago
      The issue with this is just how much context it consumes. Generating the plan could easily get me to 30% of a 1M context window—the point at which I normally clear. Most of that context fill is not necessary for execution, and the model degrades as context fills. So starting from 30% (or passing a fresh agent all of that context) is not ideal. But maybe I'm misunderstanding what you're proposing?
      - markust 3mo ago
        Yes, sorry let me explain a bit more. The idea is that we have custom agents each having a model version of their own. So, if we have a planning agent that uses Opus and a worker agent that uses Sonnet, then the planning agent writes the plan and hands it over to the worker agent. This way, the handover here is not going to retain the context. When the worker agent picks it up, it'd have a new context and only the todo list from the plan, so it shouldn't be reading all the files again.
        bensyverson 3mo ago
        Ah, got it. Yeah, that's what I do using my tool, Jobs. When the plan is big enough, it's definitely cheaper & more efficient than having the "big" agent do everything.
wrs 3mo ago
My main point was not about the implementation stage, but the verification stage. Letting the smarter agent verify the work of the cheap agent finds a lot of little oversights and sloppiness that I used to have to find myself. So I’d be interested in that cost comparison.
- bensyverson 3mo ago
  I think it would depend on how you measure it. Adding a review step will definitely increase cost, but theoretically that cost is lower than re-doing the subagent's implementation later when you discover it's broken.

ricardobeat 3mo ago

This is exactly what the post argues against though, as it leads to higher overall cost.

bensyverson 3mo ago
Yes, and I'm arguing that the post is over-simplifying. As I mention, for small plans, it's probably easier to have the big model just do it. But for large, multi-session plans, it's cheaper to let smaller models execute the bulk of the work.
- markust 3mo ago
  Perhaps, there's a small change that might work better here - if we use a different agent (by creating custom agents) for plan and implementation, then the switch to the implementation agent wouldn't end up re-reading a lot as the new agent would have fresh empty context. I implemented this[1] from one of the posters on HN and I think it'd work well with this problem too. Would be great if you could point out if my understanding is wrong. [1]: https://www.stavros.io/posts/how-i-write-software-with-llms/ https://www.stavros.io/posts/how-i-write-software-with-llms/
  - bensyverson 3mo ago
    The issue with this is just how much context it consumes. Generating the plan could easily get me to 30% of a 1M context window—the point at which I normally clear. Most of that context fill is not necessary for execution, and the model degrades as context fills. So starting from 30% (or passing a fresh agent all of that context) is not ideal. But maybe I'm misunderstanding what you're proposing?
    - markust 3mo ago
      Yes, sorry let me explain a bit more. The idea is that we have custom agents each having a model version of their own. So, if we have a planning agent that uses Opus and a worker agent that uses Sonnet, then the planning agent writes the plan and hands it over to the worker agent. This way, the handover here is not going to retain the context. When the worker agent picks it up, it'd have a new context and only the todo list from the plan, so it shouldn't be reading all the files again.
      - bensyverson 3mo ago
        Ah, got it. Yeah, that's what I do using my tool, Jobs. When the plan is big enough, it's definitely cheaper & more efficient than having the "big" agent do everything.

wrs 3mo ago

My main point was not about the implementation stage, but the verification stage. Letting the smarter agent verify the work of the cheap agent finds a lot of little oversights and sloppiness that I used to have to find myself. So I’d be interested in that cost comparison.

bensyverson 3mo ago
I think it would depend on how you measure it. Adding a review step will definitely increase cost, but theoretically that cost is lower than re-doing the subagent's implementation later when you discover it's broken.