5 ms·

I'm convinced now that the key to getting useful results out of coding agents (Claude Code, Codex CLI etc) is having good mechanisms in place to help those agen

• agumonkey 11mo ago
  gemini and claude do that already IIUC, self debugging iterations
• QuercusMax 11mo ago
  I've only done a tiny bit of agent-assisted coding, but without the ability to run tests the AI will really go off the rails super quick, and it's kinda hilarious to watch it say "Aha! I know what the problem is!" over and over as it tries different flavors until it gives up.
• formerly_proven 11mo ago
  Not so sure about formal verification though. ime with Rust LLM agents tend to struggle with semi-complex ownership or trait issues and will typically reach for unnecessary/dangerous escape hatches ("unsafe impl Send for ..." instead of using the correct locks, for example) fairly quickly. Or just conclude the task is impossible. > automatic code formatters I haven't tried this because I assumed it'll destroy agent productivity and massively increase number of tokens needed, because you're changing the file out under the LLM and it ends up constantly re-reading the changed bits to generate the correct str_replace JSON. Or are they smart enough that this quickly trains them to generate code with zero-diff under autoformatting? But in general of course anything that's helpful for human developers to be more productive will also help LLMs be more productive. For largely identical reasons.
  • simonw 11mo ago
    I'm finding my agents generate code that conforms to Black quite effectively, I think it's probably because I usually start them in existing projects that were already formatted using Black so they pick up those patterns.
    • formerly_proven 11mo ago
      I still quite often have even Opus 4.5 generate empty indented lines (regardless of explicit instructions in AGENTS.md not to (besides explicitly referencing the style guide as well), the code not containing any before and the auto-formatter removing them), for example. Trailing whitespace is much rarer but happens as well. Personally I don't care too much, since I've found LLMs to be most efficient when performing roughly the work of a handful commits at most in one thread, so I let the pre-commit hook sort it out after being done with a thread.
  • planckscnst 11mo ago
    I've directly faced this problem with automatic code formatters, but it was back around Claude 3.5 and 3.7. It would consistently write nonconforming code - regardless of having context demanding proper formatting. This caused both extra turns/invocations with the LLM and would cause context issues - both filling the context with multiple variants of the file and also having a confounding/polluting/poisoning effect due to having these multiple variations. I haven't had this problem in a while, but I expect current LLMs would probably handle those formatting instructions more closely than the 3.5 era.
• mccoyb 11mo ago
  Claude Code was a big jump for me. Another large-ish jump was multi-agents and following the tips from Anthropic’s long running harnesses post. I don’t go into Claude without everything already setup. Codex helps me curate the plan, and curate the issue tracker (one instance). Claude gets a command to fire up into context, grab an issue - implements it, and then Codex and Gemini review independently. I’ve instructed Claude to go back and forth for as many rounds as it takes. Then I close the session (\new) and do it again. These are all the latest frontier models. This is incredibly expensive, but it’s also the most reliable method I’ve found to get high-quality progress — I suspect it has something to do with ameliorating self-bias, and improving the diversity of viewpoints on the code. I suspect rigorous static tooling is yet another layer to improve the distribution over program changes, but I do think that there is a big gap in folk knowledge already between “vanilla agents” and something fancy with just raw agents, and I’m not sure if just the addition of more rigorous static tooling (beyond the compiler) closes it.
  • idiotsecant 11mo ago
    How expensive is incredibly expensive?
    • mccoyb 11mo ago
      If you're maxing out the plans across the platforms, that's 600 bucks -- but if you think about your usage and optimize, I'm guessing somewhere between 200-600 dollars per month.
      • jazzyjackson 11mo ago
        It's pretty easy to hit a couple hundred dollars a day filling up Opus's context window with files. This is via Anthropic API and Zed. Going full speed ahead building a Rails app from scratch it seemed like I was spending $50/hour, but it was worth it because the App was finished in a weekend instead of weeks. I can't bear to go in circles with Sonnet when Opus can just one shot it.
        • 11mo ago
• htrp 11mo ago
  Better question is which tools at what level
• ManuelKiessling 11mo ago
  Isn‘t it funny how that’s exactly the kind of stuff that helps a human developer be successful and productive, too? Or, to put it the other way round, what kind of tech leads would we be if we told our junior engineers „Well, here’s the codebase, that’s all I‘ll give you. No debuggers, linters, or test runners for you. Using a browser on your frontend implementation? Nice try buddy! Now good luck getting those requirements implemented!“
  • Wowfunhappy 11mo ago
    > Isn‘t it funny how that’s exactly the kind of stuff that helps a human developer be successful and productive, too? I think it's more nuanced than that. As a human, I can manually test code in ways an AI still can't. Sure, maybe it's better to have automated test suites, but I have other options too.
    • victorbjorklund 11mo ago
      AI can do that too? If you have a web app it can use playwright to test functionality and take screenshots to see if it looks right.
      • Wowfunhappy 11mo ago
        Yeah, but it doesn't work nearly as well. The AI frequently misinterprets what it sees. And it isn't as good at actually using the website (or app, or piece of hardware, etc) as a human would.
        • UncleEntity 11mo ago
          I've been using Claude to implement an ISO specification and I have to keep telling it we're not interested if the repl is correct but that the test suite is ensuring the implementation is correctly following the spec. But when we're tracking down why a test is failing then it'll go to town using the repl to narrow down out what code path is causing the issue. The only reason there's even is a repl at this point is so it can do its 'spray and pray' debugging outside the code and Claude constantly tried to use it to debug issues so I gave in and had it write a pretty basic one. Horses for courses, I suppose. Back in the day, when I wanted to play with some C(++) library, I'd quite often write a Python C-API extension so I could do the same thing using Python's repl.
• jijijijij 11mo ago
  > At the most basic level this means making sure they can run commands to execute the code Yeah, it's gonna be fun waiting for compilation cycles when those models "reason" with themselves about a semicolon. I guess we just need more compute...
• WhyOhWhyQ 11mo ago
  I've tried getting claude to set up testing frameworks, but what ends up happening is it either creates canned tests, or it forgets about tests, or it outright lies about making tests. It's definitely helpful, but feels very far from a robust thing to rely on. If you're reviewing everything the AI does then it will probably work though.
  • simonw 11mo ago
    Something I find helps a lot is having a template for creating a project that includes at least one passing test. That way the agent can run the tests at the start using the correct test harness and then add new tests as it goes along. I use cookiecutter for this, here's my latest Python library template: https://github.com/simonw/python-lib https://github.com/simonw/python-lib
  • planckscnst 11mo ago
    LLMs are very good at looking at a change set and finding untested paths. As a standard part of my workflow, I always pass the LLM's work through a "reviewer", which is a fresh LLM session with instructions to review the uncommitted changes. I include instructions for reviewing test coverage. I've also found that LLMs typically just partially implement a given task/story/spec/whatever. The reviewer stage will also notice a mismatch between the spec and the implementation. I have an orchestrator bounce the flow back and forth between developing and reviewing until the review comes back clean, and only then do I bother to review its work. It saves so much time and frustration.
    • akrauss 11mo ago
      What tooling are you using for the orchestration?
• CobrastanJorji 11mo ago
  I might go further and suggest that the key to getting useful results out of HUMAN coding agents is also to have good mechanisms in place to help them exercise and validate the code. We valued automated tests and linters and fuzzers and documentation before AI, and that's because it serves the same purpose.
  • deleted 11mo ago
    [deleted]
• oxag3n 11mo ago
  Where they'd get training data? Source code generation is possible due to large training set and effort put into reinforcing better outcomes. I suspect debugging is not that straightforward to LLM'ize. It's a non-sequential interaction - when something happens, it's not necessarily caused the problem, timeline may be shuffled. LLM would need tons of examples where something happens in debugger or logs and associate it with another abstraction. I was debugging something in gdb recently and it was a pretty challenging bug. Out of interest I tried chatgpt, and it was hopeless - try this, add this print etc. That's not how you debug multi-threaded and async code. When I found the root cause, I was analyzing how I did it and where did I learn that specific combination of techniques, each individually well documented, but never in combination - it was learning from other people and my own experience.
  • jimmaswell 11mo ago
    How long ago was this? I've had outstansingly impressive results asking Copilot Chat with Sonnet 4.5 or ChatGPT to debug difficult multithreaded C++.
    • oxag3n 11mo ago
      Few months back with ChatGPT 5. Multi-threaded Rust & custom async runtime, data integrity bug, reproduced every ~5th run.
  • simonw 11mo ago
    Have you tried running gdb from a Claude Code or Codex CLI session?
    • oxag3n 11mo ago
      No, I'm in academia and the goal is not code or product launch. I find research process to struggle a lot once someone solves a problem instead of you. I understand that AI can help with writing, coding, analyzing code bases and summarizing other papers, but going through these myself makes a difference, at least for me. I tried ChatGPT 3.5 when I started and while I got a pile of work done, I had to throw it away at some point because I didn't fully understand it. AI could explain to me various parts, but it's different when you create it.
• dionian 11mo ago
  you've done some great articles on this topic and my experience aligns with your view completely.
• zahlman 11mo ago
  One objection: all the "don't use --yolo" training in the world is useless if a sufficiently context-poisoned LLM starts putting malware in the codebase and getting to run it under the guise of "unit tests".
  • planckscnst 11mo ago
    For now, this is mitigated by only including trusted content in the context; for instance, absolutely do not allow it to access general web content. I suspect that as it becomes more economical to play with training your own models, people will get better at including obscured malicious content in data that will be used during training, which could cause the LLM to intrinsically carry a trigger/path that would cause malicious content to be output by the LLM under certain conditions. And of course we have to worry about malicious content being added to sources that we trust, but that already exists - we as an industry typically pull in public repositories without a complete review of what we're pulling. We outsource the verification to the owners of the repository. Just as we currently have cases of malicious code sneaking into common libraries, we'll have malicious content targeted at LLMs
• ramoz 11mo ago
  Claude Code hooks is a great way to integrate these things
• roadside_picnic 11mo ago
  I very much agree, and believe using languages with powerful types systems could be a big step in this direction. Most people's first experience with Haskell is "wow this is hard to write a program in, but when I do get it to compile, it works". If this works for human developers, it should also work for LLMs (especially if the human doesn't have to worry about the 'hard to write a program' part). > The next step up from that is a good automated test suite. And if we're going for a powerful type system, then we can really leverage the power of property tests which are currently grossly underused. Property tests are a perfect match for LLMs because they allow the human to create a small number of tests that cover a very wide surface of possible errors. The "thinking in types" approach to software development in Haskell allows the human user to keep at a level of abstraction that still allows them to reason about critical parts of the program while not having to worry about the more tedious implementation parts. Given how much interest there has been in using LLMs to improve Lean code for formal proofs in the math community, maybe there's a world where we make use of an even more powerful type systems than Haskell. If LLMs with the right language can help prove complex mathematical theorems, they it should certain be possible to write better software with them.
  • nextos 11mo ago
    That's my opinion as well. Some functional language, that can also offer access to imperative features when needed, plus an expressive type system might be the future. My bet is on refinement types. Dafny fits that bill quite well, it's simple, it offers refinement types, and verification is automated with SAT/SMT. In fact, there are already serious industrial efforts to generate Dafny using LLMs. Besides, some of the largest verification efforts have been achieved with this language [1]. [1] https://www.andrew.cmu.edu/user/bparno/papers/ironfleet.pdf https://www.andrew.cmu.edu/user/bparno/papers/ironfleet.pdf
  • bcrosby95 11mo ago
    Ada when? It even lets you separate implementation from specification.
    • jaggederest 11mo ago
• pron 11mo ago
  Maybe in the short term, but that doesn't solve some fundamental problems. Consider, NP problems, problems whose solutions can be easily verified. But that they can all be easily verified does not (as far as we know) mean they can all be easily solved. If we look at the P subset of NP, the problems that can be easily solved, then the easy verification is no longer their key feature. Rather, it is something else that distinguishes them from the harder problems in NP. So let's say that, similarly, there are programming tasks that are easier and harder for agents to do well. If we know that a task is in the easy category, of course having tests is good, but since we already know that an agent does it well, the test isn't the crucial aspect. On the other hand, for a hard task, all the testing in the world may not be enough for the agent to succeed. Longer term, I think it's more important to understand what's hard and what's easy for agents.
• goryDeets 11mo ago
  [dead]
• Davidzheng 11mo ago
  OK but if the verification loop really makes the agents MUCH more useful, then this usefulness difference can be used as a training signal to improve the agents themselves. So this means the current capabilities levels are certainly not going to remain for very long (which is also what I expect but I would like to point out it's also supported by this)
  • hamiecod 11mo ago
    Thats a strong RL technique that could equal the quality of RLHF.
• thomasfromcdnjs 11mo ago
  shameless plug: I'm working on an open source project https://blocksai.dev/ https://blocksai.dev/ to attempt to solve this. (and just added a note for me to add formal verification) Elevator pitch: "Blocks is a semantic linter for human-AI collaboration. Define your domain in YAML, let anyone (humans or AI) write code freely, then validate for drift. Update the code or update the spec, up to human or agent." (you can add traditional linters to the process if you want but not necessary) The gist being you define a bunch of validators for a collection of modules you're building (with agentic coding) with a focus on qualifying semantic things; - domain / business rules/measures - branding - data flow invariants — "user data never touches analytics without anonymization" - accessibility - anything you can think of Then you just tell your agentic coder to use the cli tool before committing, so it keeps the code in line with your engineering/business/philosophical values. (boring) example of it detecting if blog posts have humour in them, running in Claude Code -> https://imgur.com/diKDZ8W https://imgur.com/diKDZ8W
  • akrauss 11mo ago
    Quick feedback: both the „learn more“ link at the very top and the „Explore all examples“ link lead to 404
    • thomasfromcdnjs 11mo ago
      Thanks will fix that up shortly.
  • baq 11mo ago
    Reminder YAML is a serialization format. IaC standardizing on it (hashicorp being an outlier) was a mistake. It’s a good compilation target, but please add a higher level language for whatever you’re doing.
• rmah 11mo ago
  I think the main limitation is not code validation but assumption verification. When you ask an LLM to write some code based on a few descriptive lines of text, it is, by necessity, making a ton of assumptions. Oddly, none of the LLM's I've seen ask for clarification when multiple assumptions might all be likely. Moreover, from the behavior I've seen, they don't really backtrack to select a new assumption based on further input (I might be wrong here, it's just a feeling). What you don't specify, it must to assume. And therein lies a huge landscape of possibilities. And since the AI's can't read your mind (yet), its assumptions will probably not precisely match your assumptions unless the task is very limited in scope.
  • crazygringo 11mo ago
    > Oddly, none of the LLM's I've seen ask for clarification when multiple assumptions might all be likely. It's not odd, they've just been trained to give helpful answers straight away. If you tell them not to make assumptions and to rather first ask you all their questions together with the assumptions they would make because you want to confirm before they write the code, they'll do that too. I do that all the time, and I'll get a list of like 12 things to confirm/change. That's the great thing about LLM's -- if you want them to change their behavior, all you need to do is ask.
• rodphil 11mo ago
  > At the most basic level this means making sure they can run commands to execute the code - easiest with languages like Python, with HTML+JavaScript you need to remind them that Playwright exists and they should use it. So I've been exploring the idea of going all-in on this "basic level" of validation. I'm assembling systems out of really small "services" (written in Go) that Claude Code can immediately run and interact with using curl, jq, etc. Plus when building a particular service I already have all of the downstream services (the dependencies) built and running so a lot of dependency management and integration challenges disappear. Only trying this out at a small scale as yet, but it's fascinating how the LLMs can potentially invert a lot of the economics that inform the current conventional wisdom. (Shameless plug: I write about this here: https://twilightworld.ai/thoughts/atomic-programming/ https://twilightworld.ai/thoughts/atomic-programming/) My intuition is that LLMs will for many use cases lead us away from things like formal verification and even comprehensive test suites. The cost of those activities is justified by the larger cost of fixing things in production; I suspect that we will eventually start using LLMs to drive down the cost of production fixes, to the point where a lot of those upstream investments stop making sense.
  • jcranmer 11mo ago
    > My intuition is that LLMs will for many use cases lead us away from things like formal verification and even comprehensive test suites. The cost of those activities is justified by the larger cost of fixing things in production; I suspect that we will eventually start using LLMs to drive down the cost of production fixes, to the point where a lot of those upstream investments stop making sense. There is still a cost to having bugs, even if deploying fixes becomes much cheaper. Especially if your plan is to wait until they actually occur in practice to discover that you have a bug in the first place. Put differently: would you want the app responsible for your payroll to be developed in this manner? Especially considering that the bug in question would be "oops, you didn't get paid."
• simianwords 11mo ago
  Claude code and other AI coding tools must have a * mandatory * hook for verification. For front end - the verification is make sure that the UI looks expected (can be verified by an image model) and clicking certain buttons results in certain things (can be verified by chatgpt agent but its not public ig). For back end it will involve firing API requests one by one and verifying the results. To make this easier, we need to somehow give an environment for claude or whatever agent to run these verifications on and this is the gap that is missing. Claude Code, Codex should now start shipping verification environments that make it easy for them to verify frontend and backend tasks and I think antigravity already helps a bit here. ------ The thing about backend verification is that it is different in different companies and requires a custom implementation that can't easily be shared across companies. Each company has its own way to deploy stuff. Imagine a concrete task like creating a new service that reads from a data stream, runs transformations, puts it in another data stream where another new service consumes the transformed data and puts it into an AWS database like Aurora. ``` stream -> service (transforms) -> stream -> service -> Aurora ``` To one shot this with claude code, it must know everything about the company - how does one consume streams in the company? Schema registry? - how does one create a new service and register dependencies? how does one deploy it to test environment and production? - how does one even create an Aurora DB? request approvals and IAM roles etc? My question is: what would it take for Claude Code to one shot this? At the code level it is not too hard and it can fit in context window easily but the * main * problem is the fragmented processes in creating the infra and operations behind it which is human based now (and need not be!). ----- My prediction is that companies will make something like a new "agent" environment where all these processes (that used to require a human) can be done by an agent without human intervention. I'm thinking of other solutions here, but if anyone can figure it out, please tell!
• deleted 11mo ago
  [deleted]
• apitman 11mo ago
  That's bad news for C++, Rust, and other slow compilers.