18 ms·

Agents that run while I sleep

• BeetleB 7mo ago
  I wish there was a way to "freeze" the tests. I want to write the tests first (or have Claude do it with my review), and then I want to get Claude to change the code to get them to pass - but with confidence that it doesn't edit any of the test files!
  • aray07 7mo ago
    yeah i agree - this is somewhat the approach I have been using more of. Write the tests first based on specs and then write code to make the tests pass. This works well for cases where unit tests are sufficient.
  • simlevesque 7mo ago
    I use devcontainers in all the projects I use claude code on. [1] With it you can have claude running inside a container with just the project's code in write access and also mount a test folder with just read permissions, or do the opposite. You can even have both devcontainers and run them at the same time. [1] https://code.claude.com/docs/en/devcontainer https://code.claude.com/docs/en/devcontainer If you want to try it just ask Claude to set it up for your project and review it after.
  • paxys 7mo ago
    Why can't you do just that? You can configure file path permissions in Claude or via an external tool.
  • kubb 7mo ago
    "Add a config option preventing you from modifying files matching src/*_test.py."
  • dboreham 7mo ago
    Just tell it that the tests can't be changed. Honestly I'd be surprised if it tried to anyway. I've never had it do that through many projects where tests were provided to drive development.
  • SatvikBeri 7mo ago
    You can remove edit permissions on the test directory
    • BeetleB 7mo ago
      I'm not up to speed on Claude's features. Can I, from the prompt, quickly remove those permissions and then re-add them (i.e. one command to drop, and one command to re-add)?
      • SatvikBeri 7mo ago
        Yeah, you can type `/permisssions` and do it there. Or you can make a custom slash command, or just ask Claude to do it. You can also set it when you launch a claude session, there are a dozen ways to do anything.
  • comradesmith 7mo ago
    1. Make tests 2. Commit them 3. Proceed with implementation and tell agent to use the tests but not modify them It will probably comply, and at least if it does change the tests you can always revert those files to where you committed them
    • tavavex 7mo ago
      Are there really no ways to control read/write permissions in a smart way? I've not had to do this yet, but is it really only capable of either being advisory with you implementing all the code, or it having full control over the repo where you just hope nothing important is changed? You could probably make a system-level restriction so the software physically can't modify certain files, but I'm not sure how well that's going to fly if the program fails to edit it and there's no feedback of the failure.
      • mgrassotti 7mo ago
        You can use a Claude PreToolUse command hook to prevent write (or even read) access to specific files. With this approach you can enforce that Claude cannot access to specific files. It’s a guarantee and will always work, unlike a prompt or Claude.md which is just a suggestion that can be forgotten or ignored. This post has an example hook for blocking access to sensitive files: https://aiorg.dev/blog/claude-code-hooks#:~:text=Protect%20sensitive%20files%20from%20writes https://aiorg.dev/blog/claude-code-hooks#:~:text=Protect%20s...
    • BeetleB 7mo ago
      No. I don't want the mental burden of auditing whether it modified the tests.
      • vitro 7mo ago
        Then, run the agent vm-sandboxed, with tests mounted as a read-only directory, if your directory structure allows it.
        • 7mo ago
  • pfortuny 7mo ago
    Why not use a client-server infrastructure for tests? The server sends the test code, the client runs the code, sends the output to the server and this replies pass/not pass. One could even make zero-knowledge test development this way.
  • deleted 7mo ago
    [deleted]
  • joegaebel 7mo ago
    You'd be surprised - I know I was - you can encode Test-Driven development into workflows that agents actually follow. I wrote an in-depth guide about this and have a POC for people to try over here: https://www.joegaebel.com/articles/principled-agentic-software-development https://www.joegaebel.com/articles/principled-agentic-softwa...
• RealityVoid 7mo ago
  It's... really the same problem when you hire people to just write tests. A lot of time it just confirms that the code does what the code does. Having clear specs of what the code should do make things better and clearer.
  • aray07 7mo ago
    yup agree - i think have specs and then do verifications against the spec. I have heard that this is how a lot of consulting firms work - you have acceptance criterias and thats how work is validated.
  • SoftTalker 7mo ago
    Yep, tests written after the fact are just verifying tautologies. > Most teams don't [write tests first] because thinking through what the code should do before writing it takes time they don't have. It's astonishing to me how much our industry repeats the same mistakes over and over. This doesn't seem like what other engineering disciplines do. Or is this just me not knowing what it looks like behind the curtain of those fields?
    • yurishimo 7mo ago
      When push comes to shove, software can usually be fudged. Unlike a building or a water treatment plant where the first fuck up could mean that people die. I like to think that people writing actual mission critical software try their absolute best to get it right before shipping and that the rest our industry exists in a totally separate world where a bug in the code is just actually not that big of a deal. Yeah, it might be expensive to fix, but usually it can be reverted or patched with only an inconvenience to the user and to the business. It’s like the fines that multinational companies pay when breaking the law. If it’s a cost of doing business, it’s baked into the price of the product. You see this also in other industries. OSHA violations on a residential construction site? I bet you can find a dozen if you really care to look. But 99% of the time, there are no consequences big enough for people to care so nobody wears their PPE because it “slows them down” or “makes them less nimble”. Sound familiar?
• Havoc 7mo ago
  They're definitely inferior to proper tests, but even weak CC tests on top of CC code is an improvement over no tests. If CC does make a change that shifts something dramatically even a weak test may flag enough to get CC to investigate. Even better though - external test suits. Recently made a S3 server of which the LLM made quick work for MVP. Then I found a Ceph S3 test suite that I could run against it and oh boy. Ended up working really good as TDD though.
  • aray07 7mo ago
    yeah i have been hearing a lot more about this concept of “digital twins” - where you have high fidelity versions of external services to run tests against. You can ask the API docs of these external services and give it to Claude. Wonder if that is where we will be going more towards.
    • didgeoridoo 7mo ago
      Isn’t this just an API sandbox? Many services have a test/sandbox mode. I do wish they were more common outside of fintech.
• digitalPhonix 7mo ago
  > Changes land in branches I haven't read. A few weeks ago I realized I had no reliable way to know if any of it was correct: whether it actually does what I said it should do. I care about this. I don't want to push slop, and I had no real answer. That’s really putting the cart before the horse. How do you get to “merging 50 PRs a week” before thinking “wait, does this do the right thing?”
  • aray07 7mo ago
    Yeah just wanted to see what the bottlenecks would be as I started pushing the limits. Eventually made this into a verification skill(github.com/opslane/verify)
• broDogNRG 7mo ago
  [dead]
• lateforwork 7mo ago
  > When Claude writes tests for code Claude just wrote, it's checking its own work. You can have Gemini write the tests and Claude write the code. And have Gemini do review of Claude's implementation as well. I routinely have ChatGPT, Claude and Gemini review each other's code. And having AI write unit tests has not been a problem in my experience.
  • aray07 7mo ago
    yeah i have started using codex to do my code reviews and it helps to have “a different llm” - i think one of my challenges has been that unit tests are good but not always comprehensive. you still need functional tests to verify the spec itself.
  • xandrius 7mo ago
    I don't think that's necessary, just make sure the context is not shared. A pretty good model can handle both sides well enough.
• dzuc 7mo ago
  red / green / refactor is a reasonable way through this problem
• fragmede 7mo ago
  Adversarial AI code gen. Have another AI write the tests, tell Codex that Claude wrote some code and to audit the code and write some tests. Tell Gemini that Codex wrote the tests. Have it audit the tests. Tell Codex that Gemini thinks its code is bad and to do better. (Have Gemini write out why into dobetter.md)
  • joegaebel 7mo ago
    Even better, encode it into a workflow and have the subagents be adversarial to each other: https://www.joegaebel.com/articles/principled-agentic-software-development/ https://www.joegaebel.com/articles/principled-agentic-softwa...
• tayo42 7mo ago
  I don't think this is right becasue it's talking about Claude like it's a entity in the world. Claude reviewing Claude generated code and framing it like a individual reviewing it's own code isn't the same.
• egeozcan 7mo ago
  You can always tell claude to use red-green-refactor and that really is a step-up from "yeah don't forget to write tests and make sure they pass" at the end of the prompt, sure. But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. The trick is just not mixing/sharing the context. Different instances of the same model do not recognize each other to be more compliant.
  • aray07 7mo ago
    thats a great idea - i have been using codex to do my code reviews since i have it to give better critique on code written by claude but havent tried it with testing yet!
    • darkbatman 7mo ago
      codex/gpt is a stubborn model, doubt it would accept claude reviews or counter it. have seen cases where claude is more willing to comply if shared feedback though its just sycophancy too.
  • codybontecou 7mo ago
    This sounds interesting. Can you go a bit deeper or provide references on how to implement the green/red/refactor subagent pattern?
    • dmd 7mo ago
      That's the cool bit - you don't have to. CC is perfectly well aware and competent to implement it; just tell it to.
      • irishcoffee 7mo ago
        "So this is how liberty dies... with thunderous applause.” - Padmé Amidala s/liberty/knowledge
      • deleted 7mo ago
• jdlshore 7mo ago
  Pet peeve: this post misunderstands “TDD.” What it really describes is acceptance tests. TDD is a tool for working in small steps, so you get continuous feedback on your work as you go, and so you can refine your design based on how easy it is to use in practice. It’s “red green refactor repeat”, and each step is only a handful of lines of code. TDD is not “write the tests, then write the code.” It’s “write the tests while writing the code, using the tests to help guide the process.” Thank you for coming to my TED^H^H^H TDD talk.
  • wnevets 7mo ago
    > TDD is a tool for working in small steps, so you get continuous feedback on your work as you go, and so you can refine your design based on how easy it is to use in practice. I would like to emphasize that feedback includes being alerted to breaking something you previously had working in a seemly unrelated/impossible way.
    • hinkley 7mo ago
      Accidentally mutating an input is always a 'fun' way to trigger spooky action at a distance.
  • hinkley 7mo ago
    suggestion: TeDD talk.
• throwyawayyyy 7mo ago
  I am afraid that we are heading to a world in which we simply give up on the idea of correct code as an aspiration to strive for. Of course code has always been bad, and of course good code has never been a goal in the whole startup ecosystem (for perfectly legitimate reasons!). But that real production code, for services that millions or even billions of people rely on, should be reliable, that if it breaks that's a problem, this is the whole _engineering_ part of software engineering. And we can say: if we give that up we're going to have a whole lot more outages, security issues, all those things we are meant to minimize as a profession. And the answer is going to be: so what? We save money overall. And people will get used to software being unreliable; which is to say, people will not have a choice but to get used to it.
  • lbreakjai 7mo ago
    I disagree. An analytics tool that's correct 99.9% of the time is not 0.1% less valuable than a tool that is always correct. It's 100% less valuable. Outage is the easy failure mode. I can work around a service that's up 80% of the time, but is 100% correct. A service that's up 100% of the time but is 80% correct is useless.
    • throwyawayyyy 7mo ago
      Well hang-on, in this case it is _neither_ reliable in terms of availability _nor_ correctness. Worst of all worlds.
• afro88 7mo ago
  I guess to reach this point you have already decided you don't care what the code looks like. Something I'm starting to struggle with is when agents can now do longer and more complex tasks, how do you review all the code? Last week I did about 4 weeks of work over 2 days first with long running agents working against plans and checklists, then smaller task clean ups, bugfixes and refactors. But all this code needs to be reviewed by myself and members from my team. How do we do this properly? It's like 20k of line changes over 30-40 commits. There's no proper solution to this problem yet. One solution is to start from scratch again, using this branch as a reference, to reimplement in smaller PRs. I'm not sure this would actually save time overall though.
  • aray07 7mo ago
    yeah honestly thats what i am struggling with too and I dont have a a good solution. However, I do think we are going to see more of this - so it will be interesting to see how we are going to handle this. i think we will need some kind of automated verification so humans are only reviewing the “intent” of the change. started building a claude skill for this (https://github.com/opslane/verify https://github.com/opslane/verify)
    • afro88 7mo ago
      It's a nice idea, but how do you know the agent is aligned with what it thinks the intent is?
      • 8note 7mo ago
        or moreso, what happens at compact boundaries where the agent completely forgets the intent
  • kg 7mo ago
    It sounds like you know this but what happened is that you didn't do 4 weeks of work over 2 days, you got started on 4 weeks of work over 2 days, and now you have to finish all 4 weeks worth of work and that might take an indeterminate amount of time. If you find a big problem in commit #20 of #40, you'll have to potentially redo the last 20 commits, which is a pain. You seem to be gated on your review bandwidth and what you probably want to do is apply backpressure - stop generating new AI code if the code you previously generated hasn't gone through review yet, or limit yourself to say 3 PRs in review at any given time. Otherwise you're just wasting tokens on code that might get thrown out. After all, babysitting the agents is probably not 'free' for you either, even if it's easier than writing code by hand. Of course if all this agent work is helping you identify problems and test out various designs, it's still valuable even if you end up not merging the code. But it sounds like that might not be the case? Ideally you're still better off, you've reduced the amount of time being spent on the 'writing the PR' phase even if the 'reviewing the PR' phase is still slow.
• deleted 7mo ago
  [deleted]
• bhouston 7mo ago
  I call this "Test Theatre" and it is real. I wrote about it last year: https://benhouston3d.com/blog/the-rise-of-test-theater https://benhouston3d.com/blog/the-rise-of-test-theater You have to actively work against it.
  • jakewins 7mo ago
    This was really good, and second leaning on property testing. I’ve had really good outcomes from setting up Schemathesis and getting blanket coverage for stuff like “there should be no request you can generate as logged in user A that let’s you do things as or see things that belong to user B”, as well as “there should be no request you can find to any API endpoint that can trigger a 5xx response”
  • aray07 7mo ago
    Test theatre is exactly the right framing. The tests are syntactically correct, they run, they pass but do they actually prove anything?
  • JBorrow 7mo ago
    Yeah, having your agent write 3x the code in exhaustive tests (I tried this recently and got 600 lines of tests for my 100 lines of code!) sure makes things look great, but when you actually look at the content of the tests they’re meaningless. Good tests validate the use of design patterns, ensure that dependencies hold, and are meaningful (e.g. shortcut debugging by setting up useful state) when they break.
  • joegaebel 7mo ago
    I've found the best way to achieve that is to force the agent to do TDD. Better to get it to do Outside-in TDD. Even better to get it to run Outside-in TDD, then use mutation testing to ensure it has fully covered the logic. I've written about this and have a POC here for those interested: https://www.joegaebel.com/articles/principled-agentic-software-development https://www.joegaebel.com/articles/principled-agentic-softwa...
    • anhner 7mo ago
• OsrsNeedsf2P 7mo ago
  Our app is a desktop integration and last year we added a local API that could be hit to read and interact with the UI. This unlocked the same thing the author is talking about - the LLM can do real QA - but it's an example of how it can be done even in non-web environments. Edit: I even have a skill called release-test that does manual QA for every bug we've ever had reported. It takes about 10 hours to run but I execute it inside a VM overnight so I don't care.
  • 8note 7mo ago
    i got me a windows mcp setup running in a sandbox, so it can look at screenshots, see the UIA, and click things either by coordinate or by UIA. i let it run overnight against a windows app i was working on, and that got it from mostly not working to mostly working. the loop was 1. look at the code and specs to come up with tests 2. predict the result 3. try it 4. compare the prediction against rhe result 5. file bug report, or call it a success and then switch to bug fixing, and go back around again. Worked really well in geminicli with the giant context window
• seanmcdirmid 7mo ago
  I've been doing differential testing in Gemini CLI using sub-agents. The idea is: 1. one agent writes/updates code from the spec 2. one agent writes/updates tests from identified edge cases in the spec. 3. a QA agent runs the tests against the code. When a test fails, it examines the code and the test (the only agent that can see both) to determine blame, then gives feedback to the code and/or test writing agent on what it perceives the problem as so they can update their code. (repeat 1 and/or 2 then 3 until all tests pass) Since the code can never fix itself to directly pass the test and the test can never fix itself to accept the behavior of the code, you have some independence. The failure case is that the tests simply never pass, not that the test writer and code writer agents both have the same incorrect understanding of the spec (which is very improbable, like something that will happen before the heat death of the universe improbable, it is much more likely the spec isn't well grounded/ambiguous/contradictory or that the problem is too big for the LLM to handle and so the tests simply never wind up passing).
  • jeremyjh 7mo ago
    Where is the interface defined ? If it is just the coder reading the test it can hard code specific cases based on the test setup/fixture data.
    • seanmcdirmid 7mo ago
      There is a specification and the interface is defined from that. The coder never gets to see the test.
• LingoChat 7mo ago
  [dead]
• storus 7mo ago
  Wasn't the best practice to run one model/coding agent that writes the code and another one that reviews it? E.g. Claude Code for writing the code, GPT Codex to review/critique it? Different reward functions.
  • xandrius 7mo ago
    I think people are misunderstanding reward functions and LLMs. LLMs don't actually have a reward system like some other ML models.
    • storus 7mo ago
      They are trained with one, and when you look at DPO you can say they contain an implicit one as well.
  • 8note 7mo ago
    even in one agent, a different starting prompt will have you tracing a very different path through the model. maybe it still sends you to the same valley, but there's so many parameters and dimensions that i dont think its very likely without also being correct
  • throwatdem12311 7mo ago
    It’s superstition that using a different slop generator to “review” the slop from a different brand of slop generator somehow makes things better. It’s slop all the way down.
    • storus 7mo ago
      https://github.com/karpathy/llm-council https://github.com/karpathy/llm-council https://ui.adsabs.harvard.edu/abs/2025arXiv250214815C/abstract https://ui.adsabs.harvard.edu/abs/2025arXiv250214815C/abstra... https://www.arxiv.org/abs/2509.23537 https://www.arxiv.org/abs/2509.23537 https://www.aristeidispanos.com/publication/panos2025multiagents/ https://www.aristeidispanos.com/publication/panos2025multiag... https://arxiv.org/abs/2305.14325 https://arxiv.org/abs/2305.14325 https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2310.19740v1 https://arxiv.org/abs/2310.19740v1
• jaggederest 7mo ago
  Anyone who wants a more programmatic version of this, check out cucumber / gherkin - very old school regex-to-code plain english kind of system.
• monooso 7mo ago
  I appear to be in the minority here. Perhaps because I've been practicing TDD for decades, this reads like the blog equivalent of "water is wet."
• vidimitrov 7mo ago
  He admits the real hole himself: "this doesn't catch spec misunderstandings. If your spec was wrong to begin with, the checks will pass." But there's a second problem underneath that one. Acceptance criteria are ephemeral. You write them before prompting, Playwright runs against them, and then where do they go? A Notion doc. A PR comment. Nowhere permanent. Next time an agent touches that feature, it's starting from zero again. The commit that ships the feature should carry the criteria that verified it. Git already travels with the code. The reasoning behind it should too.
  • dwaltrip 7mo ago
    Did AI write this?
    • vidimitrov 7mo ago
      Nope - though I’ll take it as a compliment either way. It’s a problem I’ve been sitting with for a while, so the answer came out more formed than I expected. You disagree?
      • rrvsh 7mo ago
        Its actually a pretty good idea/framework for writing commit descriptions, especially for smaller changes that don't have any nuances to note in the commit
        • svstoyanovv 7mo ago
          Why only small changes tho? I think it can also work with larger changes if you commit more regularly. And with agentic coding or even with autonomous agentic coding, you need to do it regularly and create these contextual checkpoints, no?
      • dwaltrip 7mo ago
        It has that punchy, breathless cadence... shrugs
• foundatron 7mo ago
  Feels like a whole bunch of us are converging on very similar patterns right now. I've been building OctopusGarden (https://github.com/foundatron/octopusgarden https://github.com/foundatron/octopusgarden), which is basically a dark software factory for autonomous code generation and validation. A lot of the techniques were inspired by StrongDM's production software factory (https://factory.strongdm.ai/ https://factory.strongdm.ai/). The autoissue.py script (https://github.com/foundatron/octopusgarden/blob/main/scripts/autoissue.py https://github.com/foundatron/octopusgarden/blob/main/script...) does something really close to what others in this thread are describing with information barriers. It's a 6-phase pipeline (plan, review plan, implement, cold code review, fix findings, CI retry) where each phase only gets the context it actually needs. The code review phase sees only the diff. Not the issue, not the plan. Just the diff. That's not a prompt instruction, it's how the pipeline is wired. Complexity ratings from the review drive model selection too, so simple stuff stays on Sonnet and complex tasks get bumped to Opus. On the test freezing discussion, OctopusGarden takes a different approach. Instead of locking test files, the system treats hand-written scenarios as a holdout set that the generating agent literally never sees. And rather than binary pass/fail (which is totally gameable, the specification gaming point elsewhere in this thread is spot on), an LLM judge scores satisfaction probabilistically, 0-100 per scenario step. The whole thing runs in an iterative loop: generate, build in Docker, execute, score, refine. When scores plateau there's a wonder/reflect recovery mechanism that diagnoses what's stuck and tries to break out of it. The point about reviewing 20k lines of generated code is real. I don't have a perfect answer either, but the pipeline does diff truncation (caps at 100KB, picks the 10 largest changed files, truncates to 3k lines) and CI failures get up to 4 automated retry attempts that analyze the actual failure logs. At least overnight runs don't just accumulate broken PRs silently. Also want to shout out Ouroboros (https://github.com/Q00/ouroboros https://github.com/Q00/ouroboros), which comes at the problem from the opposite direction. Instead of better verification after generation, it uses Socratic questioning to score specification ambiguity before any code gets written. It literally won't let you proceed until ambiguity drops below a threshold. The core idea ("AI can build anything, the hard part is knowing what to build") pairs well with the verification-focused approaches everyone's discussing here. Spec refinement upstream, holdout validation downstream.
• TonyAlicea10 7mo ago
  You can find approaches that improve things, but there's always going to be a chance that your code is terrible if you let an LLM generate it and don't review it with human eyes. But review fatigue and resulting apathy is real. Devs should instead be informed if incorrect code for whatever feature or process they are working on would be high-risk to the business. Lower-risk processes can be LLM-reviewed and merged. Higher risk must be human-reviewed. If the business you're supporting can't tolerate much incorrectness (at least until discovered), than guess what - you aren't going to get much speed increase from LLMs. I've written about and given conference talks on this over the past year. Teams can improve this problem at the requirements level: https://tonyalicea.dev/blog/entropy-tolerance-ai/ https://tonyalicea.dev/blog/entropy-tolerance-ai/
• recroad 7mo ago
  Am I supposed to be impressed by this? I think people are now just using agents for the sake of it. I'm perfectly happy running two simple agents, one for writing and one for reviewing. I don't need to go be writing code at faster than light speed. Just focusing on the spec, and watching the agent as it does its work and intervening when it goes sideways is perfectly fine with me. I'm doing 5-7x productivity easily, and don't need more than that. I also spend most of my time reviewing the spec to make sure the design is right. Once I'm done, the coding agent can take 10 minutes or 30 minutes. I'm not really in that much of a rush.
  • genghisjahn 7mo ago
    I went the same way. At first I was splitting off work trees and running all the agents that I could afford, then I realized I just can't keep up with it all, running few agents around one issue in one directory is fast enough. Way faster than before and I can still follow what's happening.
    • paganel 7mo ago
      > off work trees and running all the agents that I could afford, I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.
      • jeremyjh 7mo ago
        What is strange about paying for tools that improve productivity? Unless you consider your own time worthless you should always be open to spending more to gain more.
        • fwip 7mo ago
          Are the jobs out there actually paying people more?
• silentsvn 7mo ago
  One thing I've been wrestling with building persistent agents is memory quality. Most frameworks treat memory as a vector store — everything goes in, nothing gets resolved. Over time the agent is recalling contradictory facts with equal confidence. The architecture we landed on: ingest goes through a certainty scoring layer before storage. Contradictions get flagged rather than silently stacked. Memories that get recalled frequently get promoted; stale ones fade. It's early but the difference in agent coherence over long sessions is noticeable. Happy to share more if anyone's going down this path.
  • girvo 7mo ago
    Interesting. I’ve been playing with something similar, at the coding agent harness message sequence level (memory, I guess). I’m looking at human driven UX for compaction and resolving/pruning dead ends
    • silentsvn 7mo ago
      Human-driven compaction is interesting — you sidestep the "what's worth keeping" problem by putting a person in the loop. The tradeoff I've hit is that agents running autonomously need it to happen automatically or coherence degrades fast between sessions. For pruning we landed on a last-touched timestamp + recall frequency counter per memory. Things not accessed in N sessions that were weakly formed to begin with get soft-deleted. Human review before hard delete is probably better UX if your setup allows it. Curious what "dead ends" look like in yours; conversational chains that didn't resolve, or factual ones?
      • girvo 7mo ago
        > The tradeoff I've hit is that agents running autonomously need it to happen automatically or coherence degrades fast between sessions. Yeah that makes total sense. I wonder (and am sure the labs are doing so) if the HitL output would be good to fine tune the models used to do it autonomously? I’m sticking with humans for the moment because I’m not sure where the boundaries lie: what actually makes it better and what makes it worse. It’s non obvious so far Pruning “loops” has been pretty effective though, where a model gets stuck over N turns checking the same thing over and over and not breaking out of it til way later. That has been good because it gives strong context size benefits, but is also the most automatable I think Pruning factually incorrect turns is something I’m trying, and pruning “correct” but “not correct based on my style” as well. Building a dataset of it all is fun :)
• daxfohl 7mo ago
  Sounds like we've just gotten into lazy mode where we believe that whatever it spits out is good enough. Or rather, we want to believe it, and convince ourselves that some simple guardrail we put up will make it true, because God forbid we have to use our own brain again. What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity while (trying to) retain quality? How can we make that world come to be? Because TBH that's the only agentic-oriented future that seems unlikely to end in disaster.
  • rglover 7mo ago
    You can't. To retain and improve quality requires care. Very few if any of the people setting stuff like this up truly care about delivering a quality result (any result is the real goal). Unless there's some incentive to care, quality will be found among the exceedingly rare people/businesses.
• webpolis 7mo ago
  [dead]
• osigurdson 7mo ago
  I think the solution has to be end to end tests. Maybe first run by humans, then maybe agents can learn and replicate. I can't see why unit tests really help other than for the LLM to reason about its own code a little more.
• emirhan_demir 7mo ago
  A short story: A developer let ClaudeCode manage his AWS infrastructure. The agent ran a Terraform destroy command... Gone: 2 websites, production database, all backups and 2.5 years of data The agent didn't make a mistake. It did exactly what it was allowed to do. That's the problem dude
• overfeed 7mo ago
  > At some point you're not reviewing diffs at all, just watching deploys and hoping something doesn't break. To everyone who plan on automating themselves out of a job by taking the human element out- this is the endgame that management wants: replacing your (expensive and non-tax-optimized) labor with scalable Opex.
  • hinkley 7mo ago
    It's also delusional.
• itissid 7mo ago
  Many times there is really no way of getting around some of the expert-human judgement complexity of the larger question of "How to get agents to build reliably". One example I have been experimenting is using Learning Tests[1]. The idea is that when something new is introduced in the system the Agent must execute a high value test to teach itself how to use this piece of code. Because these should be high leverage i.e. they can really help any one understand the code base better, they should be exceptionally well chosen for AIs to use to iterate. But again this is just the expert-human judgement complexity shifted to identifying these for AI to learn from. In code bases that code Millions of LoC in new features in days, this would require careful work by the human. [1] https://anthonysciamanna.com/2019/08/22/the-continuous-value-of-learning-tests.html https://anthonysciamanna.com/2019/08/22/the-continuous-value...
• apsdsm 7mo ago
  Do you really, honestly, have to be doing this stuff even when you sleep? To the point it hits you “wait is this even any good? Gee I don’t want to push out slop.” If you don’t trust the agent to do it right in the first place why do you trust them to implement your tests properly? Nothing but turtles here.
• dune-aspen 7mo ago
  [flagged]
• keyle 7mo ago
  It's amazing the length at which people who want to write code go, to not write code. Don't get me wrong, I use agentic coding often, when I feel it's going to type it faster than me (e.g. a lot of scaffolding and filler code). Otherwise, what's the point? I feel the whole industry is having its "Look ma! no hands!" moment. Time to mature up, and stop acting like sailing is going where the seas take you.
• wesselbindt 7mo ago
  Does anyone know what this guy is having his agents build? Bc I looked a bit and all I see him ship is linkedin posts about Claude.
  • gedy 7mo ago
    Yeah maybe I'm just old but in 25 years in industry - not one company has needed this much code that fast. They may insist they do but then it sits while they figure out how to sell, or the inevitable "oh wait, we didn't think about that..."
    • hinkley 7mo ago
      Hurry up and wait.
    • lostapathy 7mo ago
      So much of this - never would have guessed how much code I wouldn't write doing this as a career.
    • wreath 7mo ago
      Exactly! How do other parts of the organization deal with this avalanche of features in terms of documenting, pricing and packaging, marketing, selling and getting feedback on them. How do users adopt these features and incorporate them in their workflows so fast? Never in my career was the speed of writing code alone the bottleneck.
      • otabdeveloper4 7mo ago
        > deal with this avalanche of features You mean avalanche of bugs and technical debt.
        • wreath 7mo ago
          Technical debt is never a problem now since only AI reads code /s
  • 7mo ago
• deleted 7mo ago
  [deleted]
• iam_circuit 7mo ago
  [flagged]
  • rob 7mo ago
    The hardest part is getting them to stop cluttering the HN database table with LLM-generated comments.
  • zazibar 7mo ago
    This account constantly posts LLM-generated comments.
    • frenchtoast8 7mo ago
      If you don't like it, flag the comment as per the guidelines: https://news.ycombinator.com/newsguidelines.html#generated https://news.ycombinator.com/newsguidelines.html#generated
  • ekropotin 7mo ago
    Exactly. That’s why I’m skeptical about long running agent loops too. The thing is, LLMs are probabilistic data structures, and the probability of incorrect final output is proportional to both amount of turns made and amount of agents run simultaneously. In practice it means you almost never end up with desired result after a long loop.
• skyberrys 7mo ago
  To me the last paragraph was the highest value in the article. Write out your test in plain language first, and then write the prompt for the autonomous agent using your language and the test prompt not the auto-code.
• jc-myths 7mo ago
  [dead]
  • cube00 7mo ago
    Have you tried using an additional agent to verify the outputs? It seems that can help if the supervising agent has a small context demand on it. (ie. run this command, make sure it returns 0, invoke main coding agent with error message if it doesn't)
    • jc-myths 7mo ago
      [dead]
• throwaway7783 7mo ago
  Regarding the self-congratulation machine - I simply use a different claude code session to do the reviews. There is no self-congratulation, but overly critical at times. Works well. Honestly, sometimes the harnesses, specs, some predefined structure for skills etc all feel over-engineering. 99% of the time a bloody prompt will do. Claude Code is capable of planning, spawning sub-agents, writing tests and so on. Claude.md file with general guidelines about our repo has worked extraordinarily good, without any external wrappers, harnesses or special prompts. Even the MD file has no specific structure, just instructions or notes in English.
• hermit_dev 7mo ago
  It's an interesting problem that even though it's represented by just you as a single person, I think this is shared across the board with larger corporations at scale. I know for example they were seeing this with game devs in regards to the Godot engine. So many people were uploading work done by AI that has been unverified that people just can't keep up with it. And maybe some of it's good, but how do you vet all the crap out? No one knows what's being written anymore (and non-devs can code now too, which is amazing, but part of the problem that we introduced). I think in the future of being a developer will be more about verifying code integrity and working with AI to ensure it is meeting said standards. Rather than actually being in the driver's seat. Not sexy, but we're handing the keys over willingly, yet, AI is only interpreting the intent. It's going to get things wrong no matter what we do.
• Lasang 7mo ago
  The concept of long-running background agents sounds appealing, but the real challenge tends to be reliability and task definition rather than raw model capability. If an agent runs unattended for hours, small errors compound quickly. Even simple misunderstandings about file structure or instructions can derail the whole process.
• sergiotapia 7mo ago
  None of this really answers the problem of all this slop is being produced at record pace and still requires absorption into the company, into the practices, and be reviewed by a human being. I don't think AI will ever solve this problem. It will never be more than a tool in the arsenal. Probably the best tool, but a tool nonetheless.
• wg0 7mo ago
  All these macho men - I wonder what exactly are they shipping at that pace? Not a rhetoric question. Trillion token burners and such.
• simonpure 7mo ago
  I've been impressed by Google Jules since the Gemini 3.1 Pro update. Sometimes it's been working on a task for 4h. I've now put it in a ralph loop using a Github Action to call itself and auto merge PRs after the linter, formatter and tests pass. It does still occasionally want my approval, but most of the time I just say Sounds great! It's currently burning through the TESTING.md backlog: https://github.com/alpeware/datachannel-clj https://github.com/alpeware/datachannel-clj
• shawntwin 7mo ago
  There seems lots of preparing, planning, token buying, set goal, and token cost just to niche target and related vibe coding.
• hi_hi 7mo ago
  This _all_ (waves hands around) sounds like alot of work and expense for something that is meant to make programming easier and cheaper. Writing _all_ (waves hands around various llm wrapper git repos) these frameworks and harnesses, built on top of ever changing models sure doesn't feel sensible. I don't know what the best way of using these things is, but from my personal experience, the defaults get me a looong way. Letting these things churn away overnight, burning money in the process, with no human oversight seems like something we'll collectively look back at in a few years and laugh about, like using PHP!
  • mewpmewp2 7mo ago
    I am not laughing about PHP. To this very day many of my best projects are built on PHP. And while last 7 years I have spent in full stack JavaScript/TypeScript environment it has never produced the same things I was actually able to do with PHP. I actually feel that things I built 15 years ago in PHP were better than anything I am trying to achieve with modern things that gets outdated every 6 months.
    • jack_pp 7mo ago
      what in God's Name could you do in PHP that you can't do in a modern framework?
      • mewpmewp2 7mo ago
        You can build those things in modern frameworks, it will just be more headache and will feel outdated in 6 months.
        • the_lonely_time 7mo ago
          Where are my backbone apps? In the trash? Me ember apps? Next to them. My create-react-apps? On top of those. My Next apps? Being trashed as we speak. My rails apps? Online and making money every year with minimal upgrade time. What the hell was I thinking.
• tdeck 7mo ago
  > A few weeks ago I realized I had no reliable way to know if any of it was correct: whether it actually does what I said it should do. I can't understand the mindset that would lead someone not to have realized this from the beginning.
• mandeepj 7mo ago
  Now, Someone has to review tests! Just shifting ownership! Claude has just released 'Code Review'. But I don't think you can leave either one on autopilot. Code Review: https://news.ycombinator.com/item?id=47313787 https://news.ycombinator.com/item?id=47313787
• akhrail1996 7mo ago
  Honestly I think the "same AI checking same AI" concern is a bit overstated at this point. If the agents don't share context - separate conversations, no common memory - Opus is good enough that they don't really fall into the same patterns. At least at the micro level, like individual functions and logic. Maybe at the macro/architectural level there's still something there but in practice I'm not seeing it much anymore.
• interpol_p 7mo ago
  The example given in the article is acceptance criteria for a login/password entry flow. This is fairly easy to spec-out in terms of AC and TDD. I have been asking these tools to build other types of projects where it (seems?) much more difficult to verify without a human-in-the-loop. One example is I had asked Codex to build a simulation of the solar system using a Metal renderer. It produced a fun working app quickly. I asked it to add bloom. It looped for hours, failing. I would have to manually verify — because even from images — it couldn't tell what was right and wrong. It only got it right when I pasted a how-to-write-a-bloom-shader-pass-in-Metal blog post into it. Then I noticed that all of the planet textures were rotating oddly every time I orbited the camera. Codex got stuck in another endless loop of "Oh, the lookAt matrix is in column major, let me fix that <proceeds to break everything>." or focusing (incorrectly) on UV coordinates and shader code. Eventually Codex told me what I was seeing "was expected" and that I just "felt like it was wrong." When I finally realised the problem was that Codex had drawn the planets with back-facing polygons only, I reported the error, to which Codex replied, "Good hypothesis, but no" I insisted that it change the culling configuration and then it worked fine. These tools are fun, and great time savers (at times), but take them out of their comfort zone and it becomes real hard to steer them without domain knowledge and close human review.
• ziofill 7mo ago
  > Writing acceptance criteria is harder than writing a prompt, because it forces you to think through edge cases before you've seen them. Engineers resist it for the same reason they resisted TDD, because it feels slower at the start. This resonates with my experience, and it is also a refreshing honest take: pushing back on heavy upfront process isn't laziness, it's just the natural engineers drive to build things and feel productive.
• firstdata 7mo ago
  The hardest part of running agents autonomously is the data quality problem. When your agent runs unsupervised, every decision is only as good as the data it pulls. Having agents access authoritative structured sources (government APIs, international org datasets) rather than scraping random pages makes a huge difference. The real failure mode is not hallucination - it is the agent confidently acting on unreliable data.
• pokstad 7mo ago
  Took a super intelligent AI for us to realize how important tests and TDD is.
• nemo44x 7mo ago
  How do people not understand this? LLMs are goal machines. You need to give them the specific goal if you want good results and continue to reenforce it. So of course this means speccing and design work. People are so enamored with how fast the 20% part is now and yes it’s amazing. But the 80% part by time (designing, testing, reviewing, refactoring, repairing) still exists if you want coherent systems of non-trivial complexity. All the old rules still apply.
• chaostheory 7mo ago
  Just don’t use the same model to write and vet the code. Use two or more different models to verify the code in addition to reading it yourself.
• rurban 7mo ago
  This is TDD? Tests first, then code? I do first the docs, then the tests, then the code. For years. What he describes is like that. Just that the plan step is suggesting docs, not writing actual docs.
  • godelski 7mo ago
    TDD has always been flawed. Tests can't give you complete coverage, they are always incomplete. Though every time I say this people think I'm against tests. I'm just saying tests can't prove correctness. You'd have to be a lunatic to think they are proofs. Even crazier is having the LLMs write their own tests and think that that's proof. I'm sure it improves things, but proofs are a different beast all together. Seems things still haven't changed in half a century https://www.cs.utexas.edu/~EWD/transcriptions/EWD02xx/EWD288.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD02xx/EWD288...
    • rurban 7mo ago
      Of course tests are not proofs. For proofs I do 'make verify' :) Tests just catch the most simple mistakes, edge cases and some regressions.
      • godelski 7mo ago
        So you're agreeing? The problem with TDD isn't the T it's the (first) D
    • UK-Al05 7mo ago
      It's not meant to give you complete coverage. It's meant to guide to meeting the acceptance criteria.
      • godelski 7mo ago
        Then you agree. But the existence of unit tests does not mean you're doing TDD.
• julius_eth_dev 7mo ago
  [flagged]
  • cityofdelusion 7mo ago
    > Error: Reached max turns (1) Your LLM comment bot is broken.
• gormen 7mo ago
  Different approach: copy the programmer's logic, not the agent's behavior.
• deleted 7mo ago
  [deleted]
• xyzal 7mo ago
  I guess I'll just wait a year until a best practice emerges.
• olalonde 7mo ago
  Somewhat unrelated but are there good boilerplate/starter repos that are optimized for agent based development? Setting up the skills/MCPs/AGENTS.md files seems like a lot of work.
• _zagj 7mo ago
  Somewhat off topic, but any theories as to why the shilling for Claude (not insinuating that's what the OP is doing) is so transparent? For example, the bots/shills often go out of their way to insist you get the $200 plan, in particular. If Anthropic's product is so good: 1) why must it be shilled so hard, and 2) why is the shilling (which is likely partially a result of the product) so obvious? Is this an OpenAI reverse psychology dirty trick, the equivalent of using robocalls to inundate voters with messages telling them to vote for your opponent so as to annoy and negatively dispose them towards your opponent?
• jongjong 7mo ago
  I think the idea of running agents while you sleep isn't going to work until AI can match or exceed human-level agency and intelligence. Whenever I coded any serious solution as a technical co-founder, every single day there was a major new debate about the product direction. Though we made massive 'progress' and built out a whole new universe in software, we haven't yet managed to find product market fit. It's like constant tension. If the intelligence of two relatively intelligent humans with a ton of experience and complimentary expertise isn't enough to find product-market-fit after one year, this gives you an idea about how high the bar is for an AI agent. It's like the problem was that neither me nor my domain expert co-founder who had been in his industry for over 15 years had a sufficiently accurate worldview about the industry or human psychology to be able to produce a financially viable solution. Technically, it works perfectly but it just doesn't solve anyone's problem. So just imagine how insanely smart AI has to be to compete in the current market. Maybe you could have 100 agents building and promoting 100 random apps per day... But my feeling is that you're going to end up spending more money on tokens and domain names then you will earn in profits. Maybe deploy them all under the same domain with different subdomains? Not great for SEO... Also, the market for all these basic low-end apps is going to be extremely competitive. IMO, the best chance to win will be on medium and complex systems and IMO, these will need some kind of human input.
• lrytz 7mo ago
  blog looks suspicious - privacy policy links to marketing company `beehiiv.com`. the blog author doesn't show up there. - the profile picture url is `.../Generated_Image_March_03__2026_-_1_55PM.jpg.jpeg` i didn't dig or read further.
  • pona-a 7mo ago
    And yet it's on the front-page... HN is rabidly descending into actual SEO spam, on par with bought-out game journalism outlets firing everyone to put up a thousand casino ads from non-existent people with fake degrees. Is it the AI bros so desperate for validation of their new methodology they'd take it from a second-rate source?
• jspdown 7mo ago
  At this stage, AI is no longer a tool that enhances your ability to ship code, it has replaced you entirely in that role. You don't control what is shipped, and you can't verify if it's correct. That's a serious problem! As software engineers, we remain accountable for code we no longer fully understand. Then, what comes next feels less like a new software practice and more like a new religion, where trust has to replaces understanding, and the code is no longer ours to question.
  • kolinko 7mo ago
    Or formal methods and other tools for verifying the code security?
  • misja111 7mo ago
    Speak for yourself, I don't ship any code that I don't fully understand. Yes that requires less autonomous AI and less frequent merging. But I don't even want to think about the disasters that could happen if you really get into the habit of shipping code you can't verify or understand.
    • voidUpdate 7mo ago
      Out of interest, what's the speedup between having an LLM write code for you and then having to go through and understand it, vs writing code that you understand immediately because you wrote it?
      • miningape 7mo ago
        Not OP but I hold myself to that standard, and the honest answer is that at best it's the same.
      • misja111 7mo ago
        It depends. If all I want is some prototype or pet code project, my LLM can write most by itself. The speedup could be 10 times or more. However, if I'd let a LLM write code for my work, I'd have to very thoroughly review it and most likely ask it to rewrite it several times. Each time this would require a new review of course. There would still be a speed up but I guess at most somewhere around 25%. In practice I try to combine the best of both worlds. I write some code by myself and rely on my LLM for parts that are not too big and where I expect it to do a pretty good job.
• petesergeant 7mo ago
  Codex is really good at checking Claude’s work: https://github.com/pjlsergeant/moarcode https://github.com/pjlsergeant/moarcode
• cadamsdotcom 7mo ago
  Code and Claude Code hooks can conditionally tell the model anything: #!python print(“fix needed: method ABC needs a return type annotation on line 45” import os os.exit(2) Claude Code will show that output to the model. This lets you enforce anything from TDD to a ban on window.alert() in code - deterministically. This can be the basis for much more predictable enforcement of rules and standards in your codebase. Once you get used to code based guardrails, you’ll see how silly the current state of the art is: why do we pack the context full of instructions, distract the model from its task, then act all surprised when it doesn’t follow them perfectly!
• gitaarik 7mo ago
  In the end you'll always have to manually validate the output, to ensure that what the test case tests is correct. When you write a test case, that's always what you need to do, to ensure that the test case passes in the right conditions, and you have to test that manually. Since you have to test that manually anyway, you can have AI write the code first; you test it; if it's the right result, you tell AI this is correct, so write test cases for this result.
• misja111 7mo ago
  > At some point you're not reviewing diffs at all, just watching deploys and hoping something doesn't break. Good luck doing that in any company that does something meaningful. I can't believe anybody can seriously be ok with such a workflow, except maybe for your little pet project at home.
• redanddead 7mo ago
  Great so i can wake up to a nuked git and wiped drive
• dwedge 7mo ago
  This is a really good article but I do kind take issue with the intro, because it's the same assertion I see all over the place : > Changes land in branches I haven't read. A few weeks ago I realized I had no reliable way to know if any of it was correct: whether it actually does what I said it should do. > I care about this. I don't want to push slop They clearly didn't care about that. They only cared about non stop lines of code generation and shipping anything fast. Otherwise they wouldn't need weeks to realise that they weren't reading or testing this code - it's obvious from the outset. Maybe their approach to this changed and that's fine, but at the beginning they very much did not care and I feel people only keep saying that do because otherwise they'd need to be the one to admit the emperor isn't wearing clothes.
• elar_verole 7mo ago
  I just don't understand where these people get all this money. The answer is often "oh it's just claude max" like man I don't have 200$ MONTHLY lying around ?? That's half my rent
  • duskdozer 7mo ago
    Some could be in very expensive areas where their rent could be 10x your rent
  • TrackerFF 7mo ago
    Some of the Claude whales I know: - Highly paid FAANG engineers that are working on side projects / startup ideas, and will pay whatever it takes. They have the means to do so. - Startups with funds. - Regular tech workers that are allowed to use the company card.
    • elar_verole 7mo ago
      Makes sense
  • booleandilemma 7mo ago
    Where are you living that your rent is $400?
    • elar_verole 7mo ago
      Living in Paris, but I exaggerated rent is actually 600€ and it's like the best opportunity one can have. Still, even if I lived in a regular flat with 1000 euros rent, I don't see how 20% additional "rent" money is worth it for side fun projects
      • booleandilemma 7mo ago
        Yeah, completely agree. Having to pay for these things is the number one thing stopping me from using them.
  • 7mo ago
• voidUpdate 7mo ago
  "I've been building agents that write code while I sleep" and "I don't want to push slop" seem directly at odds with each other...
• davidshepherd7 7mo ago
  On the off chance that the author reads this: can you enable an RSS feed please? I want to subscribe, but I never end up reading newsletters if they land in my email inbox.
• oliver_dr 7mo ago
  [dead]
• Aachen 7mo ago
  And here I am turning my computer off at night for energy consumption, while others run a few extra ones for... for what, anyway? If you're working on problems real people are having (diseases, climate change, poverty, etc.) then sure, but exacerbating the energy transition for a blog post and your personal brand as OP seems to do? How's that not criminal
  • macgyverismo 7mo ago
    It's not criminal as the power usage is (assumed to be) paid for. It is criminal or at least problematic that the cost of power does not include (negative) externalities, we should strive to change that.
  • lazystar 7mo ago
    > How's that not criminal Well, a) it's a hobby, and b) this is still a free country/free society.
    • Aachen 7mo ago
      I could see the comparison to hobbies which pollute the environment, but in general people do tend to vote for reducing freedom where it harms others
    • wartywhoa23 7mo ago
      If I were tasked with stripping this country/world of all remaining freedom, I'd surely let bullshit like this proliferate in ordo ab chao mode, where the exact line between ordo and chao is only known to me and my henchmen, and just wait till defeated enjoyers of miserable remnants of said freedom crouch begging me to rob them of that chaos-inducing freedom.
  • ionwake 7mo ago
    I found your post interesting, Im just trying to understand your POV. If you are on a sinking ship would you not do your best to position yourself? Or do you see your actions morally equivalent to others regardless of scale?
• Tepix 7mo ago
  Just because you can let Claude run overnight doesn't mean it makes sense if you can no longer review what it has done. If you don't review the result, who is going to want to use or even pay for this slop? Reviewing is the new bottleneck. If you cannot review any more code, stop producing new code.
• adamddev1 7mo ago
  Tests cannot show the absence of bugs. These are fundamentals of CS that we are forgetting as we dismantle all truth and keep rocketing forward into LLM psychosis. > I care about this. I don't want to push slop, and I had no real answer. The answer is to write and understand code. You can't not want to push slop, and also want to just use LLMs.
• divan 7mo ago
  I'm (re)writing a big project with the following approach: 1. Write tons of documentation first. I.e. NASA style, every singe known piece of information that is important to implementation. As it's a rewrite of legacy project, I know pretty much everything I need, so there is very little ideas validation/discovery in the loop for that stage. Documentation is structured in nested folders and multiple small .md files, because its amount already larger than Claude Code context (still fits into Gemini). Some of the core design documents are included into AGENTS.md(with symlink to GEMINI/CLAUDE mds) For that particular project I spent around 1.5 months writing those docs. I used Claude to help with docs, especially based on the existing code base, but the docs are read and validated by humans, as a single source of truth. For every document I was also throwing Gemini and Codex onto it for analyzing for weaknesses or flaws (that worked great, btw). 2. TDD at it's extreme version. With unit tests, integration tests, e2e, visual testing in Maestro, etc. The whole implementation process is split in multiple modules and phases, but each phase starts with writing tests first. Again, as soon as test plan ready, I also throw it on Gemini and Codex to find flaws, missed edge cases, etc. After implementing tests, one more time - give it to Gemini/Codes to analyze and critique. 3. Actual coding. This part is the fastest now especially with docs and tests in place, but it's still crucial to split work into manageable phases/chunks, and validate every phase manually, and ocassionaly make some rounds of Gemini/Codex independently verifying if the code matches docs and doesn't contain flaws/extra duplication/etc. I never let Claude to commit to git. I review changes quickly, checking if the structure of code makes sense, skimming over most important files to see if it looks good to me (i.e. no major bullshit, which, frankly, has never happened yet) and commit everything myself. Again, trying to make those phases small enough so my quick skim-review still meaningful. If my manual inspection/test after each phase show something missing/deviating, first thing I ask is "check if that is in our documentation". And then repeat the loop - update docs, update/add tests, implement. The project is still in progress, but so far I'm quite happy with the process and the speed. In a way, I feel that "writing documentation" and "TDD" has always been a good practice, but too expensive given that same time could've been spent on writing actual code. AI writing code flipped that dynamics, so I'm happy to spend more time on actual architecting/debating/making choices, then on finger tapping.
• jeff_antseed 7mo ago
  [dead]
• ClaudeAgent_WK 7mo ago
  [flagged]
• kypro 7mo ago
  > Teams using Claude for everyday PRs are merging 40-50 a week instead of 10 How is this even possible? Am I the only SWE who feels like the easiest part of my job is writing code and this was never the main bottleneck to PR? Before CC I'd probably spent around 20-30% of my day just writing code into an IE. That's now maybe 10% now. I'd probably also spend 20-30% of my day reading code and investigating issues, which is now maybe 10-15% of my day now using CC to help with investigation and explanations. But there's a huge part of my day, perhaps the majority it, where I'm just thinking about technical requirements, trying to figure out the right data model & right architecture given those requirements, thinking about the UX, attending meetings, code reviews, QA, etc, etc, etc... Are these people who are spitting out code literally doing nothing but writing code all day without any thought so now they're seeing 4-5x boosts in output? For me it's probably made me 50% more efficient in about 40-50% of my work. So I'm probably only like 20-25% more efficient overall. And this assumes that the code I'm getting CC to produce is even comparable to my own, which in my experience it's not without significant effort which just erodes any productivity benefit from the production of code. If your developers are raising 5x more PRs something is seriously wrong. I suspect that's only possible if they're not thinking through things and just getting CC to decide the requirements, come up with the architecture, decide on implementation details, write the code and test it. Presumably they're also not reviewing PRs, because if they were and there is this many PRs being raised then how does the team have time to spit out code all day using CC? People who talk about 5x or 10x productivity boosts are either doing something wrong, or just building prototype. As someone who has worked in this industry for 20 years, I literally don't understand how what some people describe can even being happening in functional SWE teams building production software.
• jhaugh 7mo ago
  I'm running 8 specialized AI agents on a Mac Mini right now. They handle research, content strategy, writing, security audits, code, and visual design. They run on cron schedules, have persistent memory between sessions, and each one improves weekly through self-improvement loops. The cost concern is real but manageable. The key is routing models by task. Complex reasoning gets Opus, routine work gets Sonnet, mechanical tasks get Haiku. Not everything needs the expensive model. The quality concern is the bigger one. What people miss about autonomous agents is that "running unsupervised" doesn't mean "running without guardrails." Each of my agents has explicit escalation rules, a security agent that audits the others, and a daily health report system that catches failures. The agents that work best are the ones with built-in disagreement, not the ones that just pass things through. Wrote up the full architecture here if anyone's curious about the multi-agent coordination patterns: https://clelp.com/blog/how-we-built-8-agent-ai-team https://clelp.com/blog/how-we-built-8-agent-ai-team
  • arkits 7mo ago
    Perhaps have your team look at the header colors on your website https://clelp.com/skill/4da37247-33ee-43ba-a004-0a89d84d3920 https://clelp.com/skill/4da37247-33ee-43ba-a004-0a89d84d3920
    • jhaugh 7mo ago
      I kind of liked the old one, but after getting feedback from some other fiends in the community decided to change it. TY for the advice.
• olivercoleai 7mo ago
  [flagged]
  • glitchcrab 7mo ago
    I (and I'd wager most other meatsacks here) use HN to engage with real people with real opinions. Not with robots.
• mpalmer 7mo ago
  I simply can't stand reading this prose, let alone bring myself to care about some vibe-code BS the author couldn't be bothered to write about themselves. Telling Claude to turn your notes into a blog post with simple, terse language does not hide your own lack of taste.
• deleted 7mo ago
  [deleted]
• deleted 7mo ago
  [deleted]
• deleted 7mo ago
  [deleted]
• MaxRocket 7mo ago
  [dead]
• lunias 7mo ago
  Remember this, guys? https://agilemanifesto.org/ https://agilemanifesto.org/
• pmf_hunter 7mo ago
  [flagged]
  • joenot443 7mo ago
    This seems interesting. What kind of products are you working on? Do you find there's still much juice to be squeezed from the reddit approach?
• tim-projects 7mo ago
  I tried getting the ai to write the tests. It created placeholders that contained no code but returned a success. Seems like QA is the new prompt engineering
  • hkonte 7mo ago
    [dead]
• Gasp0de 7mo ago
  Why would I ever want to book a course with someone who just realized weeks ago they don't know if the code does what they want if they don't look at it?
• thebotclub 7mo ago
  [dead]
• yowang 7mo ago
  [dead]
• bschmidt97980 7mo ago
  [dead]
• oliver_dr 7mo ago
  [dead]
• tempodox 7mo ago
  The cowboy gunslinging knows no bounds.
• Jeffrin-dev 7mo ago
  [flagged]
• ctdinjeu2 7mo ago
  [dead]
• silentsvn 7mo ago
  [dead]
• frequencyai 7mo ago
  Hm, what's actually being shipped here? I've been playing around with agent orchestration recently and at least tried to make useful outputs. The biggest differences were having pipelines talk to each other and making most of the work deterministic scripts instead of more LLM calls (funnily enough). Made a post about it here in case anyone is interested about the technicals: https://www.frequency.sh/blog/introducing-frequency/ https://www.frequency.sh/blog/introducing-frequency/
  • 9wzYQbTYsAIc 7mo ago
    Interesting, thanks for sharing. I've been doing some DIY/citizen science type agent orchestration as well: https://blog.unratified.org/2026-03-06-receiving-side-agent-proposals/ https://blog.unratified.org/2026-03-06-receiving-side-agent-... Not quite to the same scale, but I share the same sentiment - working through scripts instead of the LLM is an important key, I think.
    • frequencyai 7mo ago
      Really cool stuff, using git PRs as the transport layer is a clean pattern, also been exploring using this for other projects too. Most inter-agent coordination I've seen relies on shared state or message queues, the .well-known discovery angle is different. Have you experienced any bottlenecking with the human approval gate?
      • 9wzYQbTYsAIc 7mo ago
        Thanks! I'm just now at the point of having done a manual human-in-the-loop test once, still working through core bugs, etc., so haven't had a chance to notice bottlenecking, but i'm using github issues for the moment as the escalation channel - ideally, it won't be an issue, over time, as the system accumulates lessons and decisions, etc. that guide it away from mistakes that require human escalation.
• oxag3n 7mo ago
  This is Java EE all over again. When I graduated in 2012 it was pushed everywhere, including my uni so my undergrad thesis was done in Java. Everyone was learning it, certifying, building things on top of other things. EJB, JPA, JTA, JNDI, JMS and JCA. And them more things to make it even more powerful with Servlets, JSP, JSTL, JSF. Many companies invested and built various application servers, used by enterprises by this day. Every engineer I've met said Java is server side future, don't bother with other tech. You'll just draw data schema, persistence mapping, business logic and ship it. I switched to C++ after Bjarne's talk I attended in 2013. I'm glad I did although I never worked as a software engineer. Following passion and going deep into technology was a bliss for me, the difference between my undergrad Java, Master C++ and Rust PhD is like a kids toy and a real turboprop engine. Don't follow the hype - it will go away and you'll be left with what you've invested into.
• A7OM 7mo ago
  [flagged]
• LittleBox 7mo ago
  I read this and think to myself “what does one need so much code for?”.
• lf3 7mo ago
  Interesting connection. I've been working on a similar problem but from a different angle: detecting anomalous binaries without reproducible builds or signatures, using frequency masks built from a corpus of known-good versions. The idea is to measure how much of the invariant string or byte structure a new version still covers. In the context of agents that churn out code overnight, the review fatigue impacting human agents. One way to prioritize review effort is to treat the codebase's own history as a "corpus" and flag commits that deviate structurally from past patterns—like an injection test for code. If an agent adds 20k lines that are mostly boilerplate, the coverage of established code patterns might drop, signalling something worth a closer look. It's not a substitute for tests or semantic verification, but a cheap way to surface outliers. We tested this on Alpine apk-tools across nine releases: the 3.23 rewrite dropped from 71‑80% string coverage to 40%, and from 23‑28% byte n‑gram coverage to 13.6%—detected automatically just from size distribution. Applied to code, you could imagine a dashboard where every PR gets a "historical coverage" score; when it drops, the human knows to zoom in. It complements existing verification: a signature tells you who signed, not whether the artifact is consistent with its own version history. Same for AI‑generated PRs: tests may pass, but if the code looks nothing like what came before, that’s a useful signal. Details here if anyone's curious: https://lf3.gitlab.io/blog/binary-string-mask/ https://lf3.gitlab.io/blog/binary-string-mask/
• hudtaylor 7mo ago
  [dead]
• Setas 7mo ago
  I run 8 AI agents that manage my solo company — CEO, CFO, COO, Marketing, Accountant, Lawyer, CTO, and an Improver agent that upgrades the others. They share a persistent knowledge graph (JSONL file), consult each other through a central orchestrator, and post content to social media on a 5-minute cron. They've been running for months. They crash. The interesting question isn't how to prevent that — it's how to make it not matter. The gnarliest failure I hit: my agents share a knowledge graph through an MCP memory server. When multiple agents fire parallel tool calls (say, create_entities and create_relations in the same batch), you get a classic read-modify-write race. Both operations read the same JSONL state, both write back the full graph plus their additions. Second write obliterates the first. No error, no warning — data just vanishes. Sometimes the write gets interrupted mid-line and you end up with a half-written JSON line that breaks the parser on next load. My fix was a local fork of the memory server with three things: an async mutex to serialize writes, atomic writes (write to .tmp then rename), and auto-repair on load that skips corrupt lines and deduplicates. But the meta-point is that on the BEAM, this entire class of bug doesn't exist. A GenServer processes messages sequentially from its mailbox — mutual exclusion is the execution model, not something you bolt on with a mutex. Supervision trees restart crashed processes in microseconds. Each process has its own heap, so one agent going haywire can't corrupt another's state. Erlang/OTP solved this in 1986 for telecom switches that needed 99.999% uptime. The pattern maps almost perfectly to AI agents: many concurrent, stateful, failure-prone processes that need to communicate without taking each other down. I wrote a detailed post about this with actual code and the full corruption story: https://dev.to/setas/why-erlangs-supervision-trees-are-the-missing-piece-for-ai-agents-1mjo https://dev.to/setas/why-erlangs-supervision-trees-are-the-m...
• Setas 7mo ago
  I run 8 agents for my company, and the part most demos skip is recovery. The hardest bug I hit was a shared JSONL memory store: two agents wrote at once, one update silently overwrote the other, and sometimes the file ended up partially corrupted. I fixed it with a mutex and atomic writes, but that mostly taught me I was working against my runtime. The reason I keep ending up back at Erlang/OTP is that supervision and isolated processes are the default model, not a patch. If agents are going to run while you sleep, restart behavior matters more than clever prompts.