3 ms·

What I want is a harness that knows how to optimize this kind of thing for me.

by sanderjd 4mo ago

manojlds 4mo ago
Which is your own harness and your own evals for your tasks I guess
- sanderjd 4mo ago
  Maybe. But that sounds like a large amount of bespoke work for what seems like a common problem?
  - manojlds 4mo ago
    I was talking about enterprise agents and then realized the question is more about coding agents.
    - sanderjd 4mo ago
      Ah I see! Yes, I was talking about a coding harness, not an enterprise agent. I entirely agree with you that your suggestion of driving it via evals is the right thing for that use case!
- munk-a 4mo ago
  I don't demand a customized compiler for my code even if such a compiler could outperform gcc. There is a lot of value in focusing on correctness to an extreme degree even if the outcome might be suboptimal to something more tailored - a tool with a large customer base can justify more resources going into its maintenance.
cunningfatalist 4mo ago
You might want to check out Amp: https://ampcode.com/ https://ampcode.com/
- sanderjd 4mo ago
  I appreciate the suggestion! But it isn't clear to me, from reading their marketing site, what they bring to the table from this perspective. Can you give me a more targeted pitch?
  - usef- 4mo ago
    This page buried in their docs is a bit better than the homepage imo: https://ampcode.com/manual#why-amp https://ampcode.com/manual#why-amp I haven't used them in a while so my info may be out of date, but they tended to track whatever models were the best and auto-use them for each task (eg, one for planning, subagent for a code search, other frontier for implementing). Their CLI seemed very well thought out to make you do things "the correct way" -- for instance, `/handoff` instead of `/clear`.
    - sanderjd 3mo ago
      Thanks!
  - cunningfatalist 4mo ago
    Sorry for the late answer and the missing context. usef- is right, the manual is probably the better page to share. Amp tries to give you a plug and play experience, where you can always see the actual costs and models/effort are autoselected for you. Some of my colleagues are big fans and use it a lot. I also like it, but prefer OpenCode.
    - sanderjd 3mo ago
      Great. Amp wasn't even on my radar! Appreciate the pointer!
nl 4mo ago
In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it. They are often used for reading code though. To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases. In theory the flow works like this: - small fast models read lots of code, and pass details to the large model to write a plan - large model takes those details and writes a detailed plan - medium models write the code The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model. If it guesses, the plan usually starts to fall to bits. If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read) It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup. I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.
- sanderjd 4mo ago
  Great info, thanks!
- whatshisface 4mo ago
  > In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it. That's not something I understand very well. The less expensive models will quite happily chug away at tasks, if the codebase is well-structured (small files help a lot) and your instructions are clear. In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked. You can run small models almost 10-100x as long for the same price in dollars, which covers a lot of correction and adjustment. Why does everyone say the trade-offs are rarely worth it?
  - nl 4mo ago

What I want is a harness that knows how to optimize this kind of thing for me.

manojlds 4mo ago

Which is your own harness and your own evals for your tasks I guess

sanderjd 4mo ago
Maybe. But that sounds like a large amount of bespoke work for what seems like a common problem?
- manojlds 4mo ago
  I was talking about enterprise agents and then realized the question is more about coding agents.
  - sanderjd 4mo ago
    Ah I see! Yes, I was talking about a coding harness, not an enterprise agent. I entirely agree with you that your suggestion of driving it via evals is the right thing for that use case!
munk-a 4mo ago
I don't demand a customized compiler for my code even if such a compiler could outperform gcc. There is a lot of value in focusing on correctness to an extreme degree even if the outcome might be suboptimal to something more tailored - a tool with a large customer base can justify more resources going into its maintenance.

cunningfatalist 4mo ago

You might want to check out Amp: https://ampcode.com/ https://ampcode.com/

sanderjd 4mo ago
I appreciate the suggestion! But it isn't clear to me, from reading their marketing site, what they bring to the table from this perspective. Can you give me a more targeted pitch?
- usef- 4mo ago
  This page buried in their docs is a bit better than the homepage imo: https://ampcode.com/manual#why-amp https://ampcode.com/manual#why-amp I haven't used them in a while so my info may be out of date, but they tended to track whatever models were the best and auto-use them for each task (eg, one for planning, subagent for a code search, other frontier for implementing). Their CLI seemed very well thought out to make you do things "the correct way" -- for instance, `/handoff` instead of `/clear`.
  - sanderjd 3mo ago
    Thanks!
- cunningfatalist 4mo ago
  Sorry for the late answer and the missing context. usef- is right, the manual is probably the better page to share. Amp tries to give you a plug and play experience, where you can always see the actual costs and models/effort are autoselected for you. Some of my colleagues are big fans and use it a lot. I also like it, but prefer OpenCode.
  - sanderjd 3mo ago
    Great. Amp wasn't even on my radar! Appreciate the pointer!

nl 4mo ago

In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it. They are often used for reading code though. To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases. In theory the flow works like this: - small fast models read lots of code, and pass details to the large model to write a plan - large model takes those details and writes a detailed plan - medium models write the code The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model. If it guesses, the plan usually starts to fall to bits. If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read) It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup. I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.

sanderjd 4mo ago
Great info, thanks!
whatshisface 4mo ago
> In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it. That's not something I understand very well. The less expensive models will quite happily chug away at tasks, if the codebase is well-structured (small files help a lot) and your instructions are clear. In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked. You can run small models almost 10-100x as long for the same price in dollars, which covers a lot of correction and adjustment. Why does everyone say the trade-offs are rarely worth it?
- nl 4mo ago