8 ms·

Wasting Inferences with Aider

139 points by Stwerner 1y ago

evertedsphere 1y ago
love to see "Why It Matters" turn into the heading equivalent to "delve" in body text (although different in that the latter is a legitimate word while the former is a "we need to talk about…"–level turn of phrase)
DeathArrow 1y ago
I don't really think having an agent fleet is a much better solution than having a single agent. We would like to think that having 10 agents working on the same task will improve the chances of success 10x. But I would argue that some classes of problems are hard for LLMs and where one agent will fail, 10 agents or 100 agents will fail too. As an easy example I suggest leetcode hard problems.
- adhamsalama 1y ago
  We need The Mythical Man-Month: LLM version book.
- skeledrew 1y ago
  The fleet approach can work well particularly because: 1) different models are trained differently, even though using mostly same data (think someone who studied SWE at MIT, vs one who studied at Harvard), 2) different agents can be given different prompts, which specializes their focus (think coder vs reviewer), and 3) the context window content influences the result (think someone who's seen the history of implementation attempts, vs one seeing a problem for the first time). Put those traits in various combinations and the results will be very different from a single agent.
- regularfry 1y ago
  Nit: it doesn't 10x the chance of success, it (the chance of failure)^10.
  - eMPee584 1y ago
    neither, probably
- ghuntley 1y ago
  I'm authoring a self-compiling compiler with custom lexical tokens via LLM. I'm almost at stage 2, and approximately 50 "stdlib" concerns have specifications authored for them. The idea of doing them individually in the IDE is very unappealing. Now that the object system, ast, lexer, parser, and garbage collection have stabilized, the codebase is at a point where fanning out agents makes sense. As stage 3 nears, it won't make sense to fan out until the fundamentals are ready again/stabilised, but at that point, I'll need to fan out again. https://x.com/GeoffreyHuntley/status/1911031587028042185 https://x.com/GeoffreyHuntley/status/1911031587028042185
joshstrange 1y ago
This is a very interesting idea and I really should consider Aider in the "scriptable" sense more, I only use interactively. I might add another step after each PR is created where another agent(s?) review and compare the results (maybe have the other 2 agents review the first agents code?).
- Stwerner 1y ago
  Thanks, and having another step for reviewing each other's code is a really cool extension to this, I'll give it a shot :) Whether it works or it doesn't it could be really interesting for a future post!
  - brookst 1y ago
    Wonder if you could have the reviewer characterize any mistakes and feed those back into the coding prompt: “be sure to… be sure not to…”
IshKebab 1y ago
We're going to have no traditional programming in 2 years? Riiight. It would also be nice to see a demo where the task was something that I couldn't have done myself in essentially no time. Like, what happens if you say "tasks should support tags, and you should be able to filter/group tasks by tag"?
- Stwerner 1y ago
  Gave it a shot real quick, looks like I need to fix something up about automatically running the migrations either in the CI script or locally... But if you're curious, task was this: ---- Title: Bug: Users should be able to add tags to a task to categorize them Description: Users should be able to add multiple tags to a task but aren't currently able to. Given I am a user with multiple tasks When I select one Then I should be able to add one or many tags to it Given I am a user with multiple tasks each with multiple tags When I view the list of tasks Then I should be able to see the tags associated with each task ---- And then we ended up with: GPT-4o ($0.05): https://github.com/sublayerapp/buggy_todo_app/pull/51 https://github.com/sublayerapp/buggy_todo_app/pull/51 Claude 3.5 Sonnet ($0.09): https://github.com/sublayerapp/buggy_todo_app/pull/52 https://github.com/sublayerapp/buggy_todo_app/pull/52 Gemini 2.0 Flash ($0.0018): https://github.com/sublayerapp/buggy_todo_app/pull/53 https://github.com/sublayerapp/buggy_todo_app/pull/53 One thing to note that I've found - I know you had the "...and you should be able to filter/group tasks by tag" on the request - usually when you have a request that is "feature A AND feature B" you get better results when you break it down into smaller pieces and apply them one by one. I'm pretty confident that if I spent time to get the migrations running, we'd be able to build that request out story-by-story as long as we break it out into bite-sized pieces.
  - IanCal 1y ago
    You can have a larger model split things out into more manageable steps and create new tickets - marked as blocked or not on each other, then have the whole thing run.
- victorbjorklund 1y ago
  Wouldnt AI be perfect for those easy tasks? They still take time if you wanna do it "properly" with a new branch etc. I get lots of "can you change the padding for that component". And that is all. Is it easy? Sure. But still takes time to open the project, create a new branch, make the change, push the change, create a merge request, etc. That probably takes me 10 min. If I could just let the AI do all of them and just go in and check the merge requests and approve them it would save me time.
emorning3 1y ago
I see 'Waste Inferences' as a form of abductive reasoning. I see LLMs as a form of inductive reasoning, and so I can see how WI could extend LLMs. Also, I have no doubt that there are problems that can't be solved with just an LLM but would need abductive extensions. Same comments apply to deductive (logical) extensions to LLMs.
- namaria 1y ago
  > Also, I have no doubt that there are problems that can't be solved with just an LLM but would need abductive extensions. And we're back to expert systems.
phamilton 1y ago
Sincere question: Has anyone figured out how we're going to code review the output of an agent fleet?
- jsheard 1y ago
  Insincere answer that will probably be attempted sincerely nonetheless: throw even more agents at the problem by having them do code review as well. The solution to problems caused by AI is always more AI.
  - brookst 1y ago
    s/AI/tech
  - regularfry 1y ago
    Technically that's known as "LLM-as-judge" and it's all over the literature. The intuition would be that the capability to choose between two candidates doesn't exactly overlap with the ability to generate either one of them from scratch. It's a bit like how (half of) generative adversarial networks work.
- fxtentacle 1y ago
  You just don't. Choose randomly and then try to quickly sell the company. /s
- lsllc 1y ago
  Simple, just ask an(other) AI! But seriously, different models are better/worse at different tasks, so if you can figure out which model is best at evaluating changes, use that for the review.
  - phamilton 1y ago
    I suspect this will indeed be part of it, but it won't work with today's AIs on today's codebases. Models will improve, but also I predict code style and architecture will evolve towards something easier for machine review.
danenania 1y ago
Plandex[1] uses a similar “wasteful” approach for file edits (note: I’m the creator). It orchestrates a race between diff-style replacements plus validation, writing the whole file with edits incorporated, and (on the cloud service) a specialized model plus validation. While it sounds wasteful, the calls are all very cheap since most of the input tokens are cached, and once a valid result is achieved, other in-flight requests are cancelled. It’s working quite well, allowing for quick results on easy edits with fallbacks for more complex changes/large files that don’t feel incredibly slow. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
billmalarky 1y ago
I've been lucky enough to have a few conversations with Scott a month or so ago and he is doing some really compelling work around the AISDLC and creating a factory line approach to building software. Seriously folks, I recommend following this guy closely. There's another guy in this space I know who's doing similar incredible things but he doesn't really speak about it publicly so don't want to discuss w/o his permission. I'm happy to make an introduction for those interested just hmu (check my profile for how). Really excited to see you on the FP of HN Scott!
fxtentacle 1y ago
For me, a team of junior developers that refuse to learn from their mistakes is the fuel of nightmares. I'm stuck in a loop where every day I need to explain to a new hire why they made the exact same beginner's mistake as the last person on the last day. Eventually, I'd rather spend half an hour of my own time than to explain the problem once more... Why anyone thinks having 3 different PRs for each jira ticket might boost productivity, is beyond me. Related anime: I May Be a Guild Receptionist, But I'll Solo Any Boss to Clock Out on Time
- simonw 1y ago
  One of the (many) differences between junior developers and LLM assistance is that humans can learn from their mistakes, whereas with LLMs it's up to you as the prompter to learn from their mistakes. If an LLM screws something up you can often adjust their prompt to avoid that particular problem in the future.
  - skerit 1y ago
    > One of the (many) differences between junior developers and LLM assistance is that humans can learn from their mistakes One would think so, but I've had some developers repeat the same mistake a hundred times, where eventually they admit they just keep forgetting it. The frustration you feel when telling a human for the Xth time that we do not allow yoda-conditions in our codebase is incredibly similar to when an AI does something wrong.
  - albrewer 1y ago
    > often Often being about 30% of the time in my experience
- abc-1 1y ago
  Darn I wonder if systems could be modified so that common mistakes become less common or if documentation could be written once and read multiple times by different people.
  - danielbln 1y ago
wrs 1y ago
I’ve been using Cursor and Code regularly for a few months now and the idea of letting three of them run free on the codebase seems insane. The reason for the chat interface is that the agent goes off the rails on a regular basis. At least 25% of the time I have to hit the stop button and go back to a checkpoint because the automatic lawnmower has started driving through the flowerbed again. And paradoxically, the more capable the model gets, the more likely it seems to get random ideas of how to fix things that aren’t broken.
- barrell 1y ago
  Had a similar experience with Claude Code lately. I got a notice some credits were expiring, so I opened up Claude Code and asked it to fix all the credo errors in an elixir project (style guide enforcement). I gave it incredibly clear steps of what to run in what process, maybe 6 steps, 4 of which were individual severity levels. Within a few minutes it would as to commit code, create branches, run tests, start servers — always something new, none of which were in my instructions. It would also often run mix credo, get a list of warnings, deem them unimportant, then try to go do its own thing. It was really cool, I basically worked through 1000 formatting errors in 2 hours with $40 of credits (that I would have had no use for otherwise). But man, I can’t imagine letting this thing run a single command without checking the output
  - tekacs 1y ago
    So... I know that people frame these sorts of things as if it's some kind of quantization conspiracy, but as someone who started using Claude Code the _moment_ that it came out, it felt particularly strong. Then, it feels like they... tweaked something, whether in CC or Sonnet 3.7 and it went a little downhill. It's still very impressive, but something was lost. I've found Gemini 2.5 Pro to be extremely impressive and much more able to run in an extended fashion by itself, although I've found very high variability in how well 'agent mode' works between different editors. Cursor has been very very weak in this regard for me, with Windsurf working a little better. Claude Code is excellent, but at the moment does feel let down by the model. I've been using Aider with Gemini 2.5 Pro and found that it's very much able to 'just go' by itself. I shipped a mode for Aider that lets it do so (sibling comment here) and I've had it do some huge things that run for an hour or more, but assuredly it does get stuck and act stupidly on other tasks as well. My point, more than anything, is that... I'd try different editors and different (stronger) models and see - and that small tweaks to prompt and tooling are making a big difference to these tools' effectiveness right now. Also, different models seem to excel at different problems, so switching models is often a good choice.
tekacs 1y ago
Over the last two days, I've built out support for autonomy in Aider (a lot like Claude Code) that hybridizes with the rest of the app: https://github.com/Aider-AI/aider/pull/3781 https://github.com/Aider-AI/aider/pull/3781 Edit: In case anyone wants to try it, I uploaded it to PyPI as `navigator-mode`, until (and if!) the PR is accepted. By I, I mean that it uploaded itself. You can see the session where it did that here: https://asciinema.org/a/9JtT7DKIRrtpylhUts0lr3EfY https://asciinema.org/a/9JtT7DKIRrtpylhUts0lr3EfY Edit 2: And as a Show HN, too: https://news.ycombinator.com/item?id=43674180 https://news.ycombinator.com/item?id=43674180 and, because Aider's already an amazing platform without the autonomy, it's very easy to use the rest of Aider's options, like using `/ask` first, using `/code` or `/architect` for specific tasks [1], but if you start in `/navigator` mode (which I built, here), you can just... ask for a particular task to be done and... wait and it'll often 'just get done'. It's... decidedly expensive to run an LLM this way right now (Gemini 2.5 Pro is your best bet), but if it's $N today, I don't doubt that it'll be $0.N by next year. I don't mean to speak in meaningless hype, but I think that a lot of folks who are speaking to LLMs' 'inability' to do things are also spending relatively cautiously on them, when tomorrow's capabilities are often here, just pricey. I'm definitely still intervening as it goes (as in the Devin demos, say), but I'm also having LLMs relatively autonomously build out large swathes of functionality, the kind that I would put off or avoid without them. I wouldn't call it a programmer-replacement any time soon (it feels far from that), but I'm solo finishing architectures now that I know how to build, but where delegating them to a team of senior devs would've resulted in chaos. [1]: also for anyone who hasn't tried it and doesn't like TUI, do note that Aider has a web mode and a 'watch mode', where you can use your normal editor and if you leave a comment like '# make this darker ai!', Aider will step in and apply the change. This is even fancier with navigator/autonomy.
- nico 1y ago
  > It's... decidedly expensive to run an LLM this way right now Does it work ok with local models? Something like the quantized deepseeks, gemma3 or llamas?
  - tekacs 1y ago
    It does for me, yes -- models seem to be pretty capable of adhering to the tool call format, which is really all that they 'need' in order to do a good job. I'm still tweaking the prompts (and I've introduced a new, tool-call based edit format as a primary replacement to Aider's usual SEARCH/REPLACE, which is both easier and harder for LLMs to use - but it allows them to better express e.g. 'change the name of this function'). So... if you have any trouble with it, I would adjust the prompts (in `navigator_prompts.py` and `navigator_legacy_prompts.py` for non-tool-based editing). In particular when I adopted more 'terseness and proactively stop' prompting, weaker LLMs started stopping prematurely more often. It's helpful for powerful thinking models (like Sonnet and Gemini 2.5 Pro), but for smaller models I might need to provide an extra set of prompts that let them roam more.
aqme28 1y ago
It's cute but I don't see the benefit. In my experience, if one LLM fails to solve a problem, the other ones won't be too different. If you picked a problem where LLMs are good, now you have to review 3 PRs instead of just 1. If you picked a problem where they're bad, now you have 3 failures. I think there are not many cases where throwing more attempts at the problem is useful.
denidoman 1y ago
The current challenge is not to create a patch, but to verify it. Testing a fix in a big application is a very complex task. First of all, you have to reproduce the issue, to verify steps (or create them, because many issues don't contain clear description). Then you should switch to the fixed version and make sure that the issue doesn't exists. Finally, you should apply little exploratory testing to make sure that the fix doesn't corrupted neighbour logic (deep application knowledge required to perform it). To perform these steps you have to deploy staging with the original/fixed versions or run everything locally and do pre-setup (create users, entities, etc. to achieve the corrupted state). This is very challenging area for the current agents. Now they just can't do these steps - their mental models just not ready for a such level of integration into the app and infra. And creation of 3/5/10/100 unverified pull requests just slow down software development process.
- deleted 1y ago
  [deleted]
- gandalfgeek 1y ago
  There is no fundamental blocker to agents doing all those things. Mostly a matter of constructing the right tools and grounding, which can be fair amount of up-front work. Arming LLMs with the right tools and documentation got us this far. There’s no reason to believe that path is exhausted.
  - dimitri-vs 1y ago
    Have you tried building agents? They will go from PhD level smart to making mistakes a middle schooler would find obvious, even on models like gemini-2.5 and o1-pro. It's almost like building a sandcastle where once you get a prompt working you become afraid to make any changes because something else will break.
    - sdesol 1y ago
      > Have you tried building agents? I think the issue right now is so many people want to believe in the moonshot and are investing heavily in it, when the reality is we should be focusing on the home runs. LLMs are a game changer, but there is still A LOT of tooling that can be created to make it easier to integrate humans in the loop.
lherron 1y ago
I love this! I have a similar automation for moving a feature through ideation/requirements/technical design, but I usually dump the result into Cursor for last mile and to save on inference. Seeing the cost analysis is eye opening. There’s probably also some upside to running the same model multiple times. I find Sonnet will sometimes fail, I’ll roll back and try again with same prompt but clean context, and it will succeed.
- ghuntley 1y ago
  re: cost analysis There's something cooked about Windsurf/Cursors' go-to-market pricing - there's no way they are turning a profit at $50/month. $50/month gets you a happy meal experience. If you want more power, you gotta ditch snacking at McDonald’s. In the future, companies should budget $100 USD to $500 USD per day, per dev, on tokens as the new normal for business, which is circa $25k USD (low end) to $50k USD (likely) to $127k USD (highest) per year. Above from https://ghuntley.com/redlining/ https://ghuntley.com/redlining/ This napkin math is based upon my current spend in bring a self-compiled compiler to life.
precompute 1y ago
Feels like a way to live with a bad decision rather than getting rid of it.
pton_xd 1y ago
The trend with LLMs so far has been: if you have an issue with the AI, wait 6 months for a more advanced model. Cobbling together workarounds for their deficiencies is basically a waste of effort.
pinoy420 1y ago
[dead]
KTibow 1y ago
I wonder if using thinking models would work better here. They generally have less variance and consider more options, which could achieve the same goal.
canterburry 1y ago
I wouldn't be surprised if someone tries to leverage this with their customer feature request tool. Imagine having your customers write feature requests for your saas, that immediately triggers code generation and a PR. A virtual environment with that PR is spun up and served to that customer for feedback and refinement. Loop until customer has implemented the feature they would like to see in your product. Enterprise plan only, obviously.
kgeist 1y ago
I've noticed that large models from different vendors often end up converging on more or less the same ideas (probably because they're trained on more or less the same data). A few days ago, I asked both Grok and ChatGPT to produce several stories with an absurd twist, and they consistently generated the same twists, differing only in minor details. Often, they even used identical wording! Is there any research into this phenomenon? Is code generation any different? Isn't there a chance that several "independent" models might produce the same (say, faulty) result?
dimal 1y ago
Makes me think of The Sorcerers Apprentice.
charlie0 1y ago
The 10 cents is BS. It was only that because it was a trivial bug. A non-trivial bug requires context and the more context something requires, the more expensive it gets. Also once you are working with larger apps you have to pick the context, especially with LLMs that have smaller windows.

139 points by Stwerner 1y ago

love to see "Why It Matters" turn into the heading equivalent to "delve" in body text (although different in that the latter is a legitimate word while the former is a "we need to talk about…"–level turn of phrase)

DeathArrow 1y ago

I don't really think having an agent fleet is a much better solution than having a single agent. We would like to think that having 10 agents working on the same task will improve the chances of success 10x. But I would argue that some classes of problems are hard for LLMs and where one agent will fail, 10 agents or 100 agents will fail too. As an easy example I suggest leetcode hard problems.

adhamsalama 1y ago
We need The Mythical Man-Month: LLM version book.
skeledrew 1y ago
The fleet approach can work well particularly because: 1) different models are trained differently, even though using mostly same data (think someone who studied SWE at MIT, vs one who studied at Harvard), 2) different agents can be given different prompts, which specializes their focus (think coder vs reviewer), and 3) the context window content influences the result (think someone who's seen the history of implementation attempts, vs one seeing a problem for the first time). Put those traits in various combinations and the results will be very different from a single agent.
regularfry 1y ago
Nit: it doesn't 10x the chance of success, it (the chance of failure)^10.
- eMPee584 1y ago
  neither, probably
ghuntley 1y ago
I'm authoring a self-compiling compiler with custom lexical tokens via LLM. I'm almost at stage 2, and approximately 50 "stdlib" concerns have specifications authored for them. The idea of doing them individually in the IDE is very unappealing. Now that the object system, ast, lexer, parser, and garbage collection have stabilized, the codebase is at a point where fanning out agents makes sense. As stage 3 nears, it won't make sense to fan out until the fundamentals are ready again/stabilised, but at that point, I'll need to fan out again. https://x.com/GeoffreyHuntley/status/1911031587028042185 https://x.com/GeoffreyHuntley/status/1911031587028042185

joshstrange 1y ago

This is a very interesting idea and I really should consider Aider in the "scriptable" sense more, I only use interactively. I might add another step after each PR is created where another agent(s?) review and compare the results (maybe have the other 2 agents review the first agents code?).

Stwerner 1y ago
Thanks, and having another step for reviewing each other's code is a really cool extension to this, I'll give it a shot :) Whether it works or it doesn't it could be really interesting for a future post!
- brookst 1y ago
  Wonder if you could have the reviewer characterize any mistakes and feed those back into the coding prompt: “be sure to… be sure not to…”

IshKebab 1y ago

We're going to have no traditional programming in 2 years? Riiight. It would also be nice to see a demo where the task was something that I couldn't have done myself in essentially no time. Like, what happens if you say "tasks should support tags, and you should be able to filter/group tasks by tag"?

Stwerner 1y ago
Gave it a shot real quick, looks like I need to fix something up about automatically running the migrations either in the CI script or locally... But if you're curious, task was this: ---- Title: Bug: Users should be able to add tags to a task to categorize them Description: Users should be able to add multiple tags to a task but aren't currently able to. Given I am a user with multiple tasks When I select one Then I should be able to add one or many tags to it Given I am a user with multiple tasks each with multiple tags When I view the list of tasks Then I should be able to see the tags associated with each task ---- And then we ended up with: GPT-4o ($0.05): https://github.com/sublayerapp/buggy_todo_app/pull/51 https://github.com/sublayerapp/buggy_todo_app/pull/51 Claude 3.5 Sonnet ($0.09): https://github.com/sublayerapp/buggy_todo_app/pull/52 https://github.com/sublayerapp/buggy_todo_app/pull/52 Gemini 2.0 Flash ($0.0018): https://github.com/sublayerapp/buggy_todo_app/pull/53 https://github.com/sublayerapp/buggy_todo_app/pull/53 One thing to note that I've found - I know you had the "...and you should be able to filter/group tasks by tag" on the request - usually when you have a request that is "feature A AND feature B" you get better results when you break it down into smaller pieces and apply them one by one. I'm pretty confident that if I spent time to get the migrations running, we'd be able to build that request out story-by-story as long as we break it out into bite-sized pieces.
- IanCal 1y ago
  You can have a larger model split things out into more manageable steps and create new tickets - marked as blocked or not on each other, then have the whole thing run.
victorbjorklund 1y ago
Wouldnt AI be perfect for those easy tasks? They still take time if you wanna do it "properly" with a new branch etc. I get lots of "can you change the padding for that component". And that is all. Is it easy? Sure. But still takes time to open the project, create a new branch, make the change, push the change, create a merge request, etc. That probably takes me 10 min. If I could just let the AI do all of them and just go in and check the merge requests and approve them it would save me time.

emorning3 1y ago

I see 'Waste Inferences' as a form of abductive reasoning. I see LLMs as a form of inductive reasoning, and so I can see how WI could extend LLMs. Also, I have no doubt that there are problems that can't be solved with just an LLM but would need abductive extensions. Same comments apply to deductive (logical) extensions to LLMs.

namaria 1y ago
> Also, I have no doubt that there are problems that can't be solved with just an LLM but would need abductive extensions. And we're back to expert systems.

phamilton 1y ago

Sincere question: Has anyone figured out how we're going to code review the output of an agent fleet?

jsheard 1y ago
Insincere answer that will probably be attempted sincerely nonetheless: throw even more agents at the problem by having them do code review as well. The solution to problems caused by AI is always more AI.
- brookst 1y ago
  s/AI/tech
- regularfry 1y ago
  Technically that's known as "LLM-as-judge" and it's all over the literature. The intuition would be that the capability to choose between two candidates doesn't exactly overlap with the ability to generate either one of them from scratch. It's a bit like how (half of) generative adversarial networks work.
fxtentacle 1y ago
You just don't. Choose randomly and then try to quickly sell the company. /s
lsllc 1y ago
Simple, just ask an(other) AI! But seriously, different models are better/worse at different tasks, so if you can figure out which model is best at evaluating changes, use that for the review.
- phamilton 1y ago
  I suspect this will indeed be part of it, but it won't work with today's AIs on today's codebases. Models will improve, but also I predict code style and architecture will evolve towards something easier for machine review.

danenania 1y ago

Plandex[1] uses a similar “wasteful” approach for file edits (note: I’m the creator). It orchestrates a race between diff-style replacements plus validation, writing the whole file with edits incorporated, and (on the cloud service) a specialized model plus validation. While it sounds wasteful, the calls are all very cheap since most of the input tokens are cached, and once a valid result is achieved, other in-flight requests are cancelled. It’s working quite well, allowing for quick results on easy edits with fallbacks for more complex changes/large files that don’t feel incredibly slow. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex

billmalarky 1y ago

I've been lucky enough to have a few conversations with Scott a month or so ago and he is doing some really compelling work around the AISDLC and creating a factory line approach to building software. Seriously folks, I recommend following this guy closely. There's another guy in this space I know who's doing similar incredible things but he doesn't really speak about it publicly so don't want to discuss w/o his permission. I'm happy to make an introduction for those interested just hmu (check my profile for how). Really excited to see you on the FP of HN Scott!

fxtentacle 1y ago

For me, a team of junior developers that refuse to learn from their mistakes is the fuel of nightmares. I'm stuck in a loop where every day I need to explain to a new hire why they made the exact same beginner's mistake as the last person on the last day. Eventually, I'd rather spend half an hour of my own time than to explain the problem once more... Why anyone thinks having 3 different PRs for each jira ticket might boost productivity, is beyond me. Related anime: I May Be a Guild Receptionist, But I'll Solo Any Boss to Clock Out on Time

simonw 1y ago
One of the (many) differences between junior developers and LLM assistance is that humans can learn from their mistakes, whereas with LLMs it's up to you as the prompter to learn from their mistakes. If an LLM screws something up you can often adjust their prompt to avoid that particular problem in the future.
- skerit 1y ago
  > One of the (many) differences between junior developers and LLM assistance is that humans can learn from their mistakes One would think so, but I've had some developers repeat the same mistake a hundred times, where eventually they admit they just keep forgetting it. The frustration you feel when telling a human for the Xth time that we do not allow yoda-conditions in our codebase is incredibly similar to when an AI does something wrong.
- albrewer 1y ago
  > often Often being about 30% of the time in my experience
abc-1 1y ago
Darn I wonder if systems could be modified so that common mistakes become less common or if documentation could be written once and read multiple times by different people.
- danielbln 1y ago

wrs 1y ago

I’ve been using Cursor and Code regularly for a few months now and the idea of letting three of them run free on the codebase seems insane. The reason for the chat interface is that the agent goes off the rails on a regular basis. At least 25% of the time I have to hit the stop button and go back to a checkpoint because the automatic lawnmower has started driving through the flowerbed again. And paradoxically, the more capable the model gets, the more likely it seems to get random ideas of how to fix things that aren’t broken.

barrell 1y ago
Had a similar experience with Claude Code lately. I got a notice some credits were expiring, so I opened up Claude Code and asked it to fix all the credo errors in an elixir project (style guide enforcement). I gave it incredibly clear steps of what to run in what process, maybe 6 steps, 4 of which were individual severity levels. Within a few minutes it would as to commit code, create branches, run tests, start servers — always something new, none of which were in my instructions. It would also often run mix credo, get a list of warnings, deem them unimportant, then try to go do its own thing. It was really cool, I basically worked through 1000 formatting errors in 2 hours with $40 of credits (that I would have had no use for otherwise). But man, I can’t imagine letting this thing run a single command without checking the output
- tekacs 1y ago
  So... I know that people frame these sorts of things as if it's some kind of quantization conspiracy, but as someone who started using Claude Code the _moment_ that it came out, it felt particularly strong. Then, it feels like they... tweaked something, whether in CC or Sonnet 3.7 and it went a little downhill. It's still very impressive, but something was lost. I've found Gemini 2.5 Pro to be extremely impressive and much more able to run in an extended fashion by itself, although I've found very high variability in how well 'agent mode' works between different editors. Cursor has been very very weak in this regard for me, with Windsurf working a little better. Claude Code is excellent, but at the moment does feel let down by the model. I've been using Aider with Gemini 2.5 Pro and found that it's very much able to 'just go' by itself. I shipped a mode for Aider that lets it do so (sibling comment here) and I've had it do some huge things that run for an hour or more, but assuredly it does get stuck and act stupidly on other tasks as well. My point, more than anything, is that... I'd try different editors and different (stronger) models and see - and that small tweaks to prompt and tooling are making a big difference to these tools' effectiveness right now. Also, different models seem to excel at different problems, so switching models is often a good choice.

tekacs 1y ago

Over the last two days, I've built out support for autonomy in Aider (a lot like Claude Code) that hybridizes with the rest of the app: https://github.com/Aider-AI/aider/pull/3781 https://github.com/Aider-AI/aider/pull/3781 Edit: In case anyone wants to try it, I uploaded it to PyPI as `navigator-mode`, until (and if!) the PR is accepted. By I, I mean that it uploaded itself. You can see the session where it did that here: https://asciinema.org/a/9JtT7DKIRrtpylhUts0lr3EfY https://asciinema.org/a/9JtT7DKIRrtpylhUts0lr3EfY Edit 2: And as a Show HN, too: https://news.ycombinator.com/item?id=43674180 https://news.ycombinator.com/item?id=43674180 and, because Aider's already an amazing platform without the autonomy, it's very easy to use the rest of Aider's options, like using `/ask` first, using `/code` or `/architect` for specific tasks [1], but if you start in `/navigator` mode (which I built, here), you can just... ask for a particular task to be done and... wait and it'll often 'just get done'. It's... decidedly expensive to run an LLM this way right now (Gemini 2.5 Pro is your best bet), but if it's $N today, I don't doubt that it'll be $0.N by next year. I don't mean to speak in meaningless hype, but I think that a lot of folks who are speaking to LLMs' 'inability' to do things are also spending relatively cautiously on them, when tomorrow's capabilities are often here, just pricey. I'm definitely still intervening as it goes (as in the Devin demos, say), but I'm also having LLMs relatively autonomously build out large swathes of functionality, the kind that I would put off or avoid without them. I wouldn't call it a programmer-replacement any time soon (it feels far from that), but I'm solo finishing architectures now that I know how to build, but where delegating them to a team of senior devs would've resulted in chaos. [1]: also for anyone who hasn't tried it and doesn't like TUI, do note that Aider has a web mode and a 'watch mode', where you can use your normal editor and if you leave a comment like '# make this darker ai!', Aider will step in and apply the change. This is even fancier with navigator/autonomy.

nico 1y ago
> It's... decidedly expensive to run an LLM this way right now Does it work ok with local models? Something like the quantized deepseeks, gemma3 or llamas?
- tekacs 1y ago
  It does for me, yes -- models seem to be pretty capable of adhering to the tool call format, which is really all that they 'need' in order to do a good job. I'm still tweaking the prompts (and I've introduced a new, tool-call based edit format as a primary replacement to Aider's usual SEARCH/REPLACE, which is both easier and harder for LLMs to use - but it allows them to better express e.g. 'change the name of this function'). So... if you have any trouble with it, I would adjust the prompts (in `navigator_prompts.py` and `navigator_legacy_prompts.py` for non-tool-based editing). In particular when I adopted more 'terseness and proactively stop' prompting, weaker LLMs started stopping prematurely more often. It's helpful for powerful thinking models (like Sonnet and Gemini 2.5 Pro), but for smaller models I might need to provide an extra set of prompts that let them roam more.

aqme28 1y ago

It's cute but I don't see the benefit. In my experience, if one LLM fails to solve a problem, the other ones won't be too different. If you picked a problem where LLMs are good, now you have to review 3 PRs instead of just 1. If you picked a problem where they're bad, now you have 3 failures. I think there are not many cases where throwing more attempts at the problem is useful.

denidoman 1y ago

The current challenge is not to create a patch, but to verify it. Testing a fix in a big application is a very complex task. First of all, you have to reproduce the issue, to verify steps (or create them, because many issues don't contain clear description). Then you should switch to the fixed version and make sure that the issue doesn't exists. Finally, you should apply little exploratory testing to make sure that the fix doesn't corrupted neighbour logic (deep application knowledge required to perform it). To perform these steps you have to deploy staging with the original/fixed versions or run everything locally and do pre-setup (create users, entities, etc. to achieve the corrupted state). This is very challenging area for the current agents. Now they just can't do these steps - their mental models just not ready for a such level of integration into the app and infra. And creation of 3/5/10/100 unverified pull requests just slow down software development process.

deleted 1y ago
[deleted]
gandalfgeek 1y ago
There is no fundamental blocker to agents doing all those things. Mostly a matter of constructing the right tools and grounding, which can be fair amount of up-front work. Arming LLMs with the right tools and documentation got us this far. There’s no reason to believe that path is exhausted.
- dimitri-vs 1y ago
  Have you tried building agents? They will go from PhD level smart to making mistakes a middle schooler would find obvious, even on models like gemini-2.5 and o1-pro. It's almost like building a sandcastle where once you get a prompt working you become afraid to make any changes because something else will break.
  - sdesol 1y ago
    > Have you tried building agents? I think the issue right now is so many people want to believe in the moonshot and are investing heavily in it, when the reality is we should be focusing on the home runs. LLMs are a game changer, but there is still A LOT of tooling that can be created to make it easier to integrate humans in the loop.

lherron 1y ago

I love this! I have a similar automation for moving a feature through ideation/requirements/technical design, but I usually dump the result into Cursor for last mile and to save on inference. Seeing the cost analysis is eye opening. There’s probably also some upside to running the same model multiple times. I find Sonnet will sometimes fail, I’ll roll back and try again with same prompt but clean context, and it will succeed.

ghuntley 1y ago
re: cost analysis There's something cooked about Windsurf/Cursors' go-to-market pricing - there's no way they are turning a profit at $50/month. $50/month gets you a happy meal experience. If you want more power, you gotta ditch snacking at McDonald’s. In the future, companies should budget $100 USD to $500 USD per day, per dev, on tokens as the new normal for business, which is circa $25k USD (low end) to $50k USD (likely) to $127k USD (highest) per year. Above from https://ghuntley.com/redlining/ https://ghuntley.com/redlining/ This napkin math is based upon my current spend in bring a self-compiled compiler to life.

precompute 1y ago

Feels like a way to live with a bad decision rather than getting rid of it.

pton_xd 1y ago

The trend with LLMs so far has been: if you have an issue with the AI, wait 6 months for a more advanced model. Cobbling together workarounds for their deficiencies is basically a waste of effort.

pinoy420 1y ago

[dead]

KTibow 1y ago

I wonder if using thinking models would work better here. They generally have less variance and consider more options, which could achieve the same goal.

canterburry 1y ago

I wouldn't be surprised if someone tries to leverage this with their customer feature request tool. Imagine having your customers write feature requests for your saas, that immediately triggers code generation and a PR. A virtual environment with that PR is spun up and served to that customer for feedback and refinement. Loop until customer has implemented the feature they would like to see in your product. Enterprise plan only, obviously.

kgeist 1y ago

I've noticed that large models from different vendors often end up converging on more or less the same ideas (probably because they're trained on more or less the same data). A few days ago, I asked both Grok and ChatGPT to produce several stories with an absurd twist, and they consistently generated the same twists, differing only in minor details. Often, they even used identical wording! Is there any research into this phenomenon? Is code generation any different? Isn't there a chance that several "independent" models might produce the same (say, faulty) result?

dimal 1y ago

Makes me think of The Sorcerers Apprentice.

charlie0 1y ago

The 10 cents is BS. It was only that because it was a trivial bug. A non-trivial bug requires context and the more context something requires, the more expensive it gets. Also once you are working with larger apps you have to pick the context, especially with LLMs that have smaller windows.