4 ms·

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study tru

by onion2k 4mo ago

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to do the validation, we could follow the vibe-coder approach by skipping the validation entirely, and trust that the generation stage is good enough not to need it. Historically that's come with some small downsides, like the code being a broken mess of security holes, but with time AI might fix that.

alfalfasprout 4mo ago
small downsides like security holes? Those aren't small. Neither is creating a codebase that's an inextensible mess that even LLMs begin to struggle with. The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.
- onion2k 4mo ago
  small downsides like security holes? Those aren't small I'm British. I've been taught to turn understatement into an art.
hnlmorg 4mo ago
I wouldn’t even trust experienced developers to merge code without peer review.
- roenxi 4mo ago
  It is getting to the point where I wouldn't trust experienced developers to merge code without AI review. The latest generation of models are getting pretty good.
  - IanCal 4mo ago
    Definitely for the time and cost too. If a dev costs $1k/day that’s $30 to spend 15 minutes looking at a pr. How much review does that in tokens get you? I’d wager you’d easily find lots of low level issues that are beyond basic linting with that reliably, and I feel like latest gen models can do really good work.
- onion2k 4mo ago
  I would, but that's mostly because I don't trust PRs to catch real problems. Someone reviewing the changes in a codebase is never going to spot an architectural or code design issue, and those are the real ones you need to care about. In my experience 95% of everything that's caught by a human PR review could have been caught by a linter or a formatter before the PR was opened. If you trust your team to care about quality then PRs aren't necessary, and if you don't then why are you trusting them to catching problems in PR reviews?
  - IanCal 4mo ago
    Outside of the benefit of some extra documentation around changes and having more than one person see what’s happened there are a few main safety parts here: * It can be easier to see a problem in something that another person wrote, you’re not clouded by what you intended to write * different skill sets mean two sets of eyes broadens the kind of problems that can be found (benefits from selecting good reviewers, maybe I’m tagged because I know the llm APIs and performances better by someone who has done refactoring to improve, say, internal caching that’s more their thing) * just chance. There’s some chance you spot an issue, some chance someone else does. Combined it’s better. * I disagree they can’t find architectural or design issues. You see repeated changes of the same kind, or tying together things that shouldn’t be, etc. But yes, many things could be caught before opening. Lots I catch as I’m explaining the change, like rubber ducking. I quite like AI code reviews things for this, there’s a whole back and forth that can be avoided once you get past basic linting/test level things. Save the human time for understanding the higher level issues.
beaugunderson 4mo ago
> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would need to test it myself since MFA requires setup (and a TOTP device!) but no, the agent drove Playwright to change the instance configuration to enable MFA, created a user with MFA enabled, wrote the tiniest (handful of lines) Python TOTP and used that as the device and successfully verified that MFA still worked as expected. I did not expect that ability, and I wrote it! It was also exhaustive in its probe of functionality, way more than I had planned to do, even though I had planned to do a lot given the critical nature of MFA. Total token price for the UAT run: $5.69.
- weitendorf 4mo ago
  I built a very similar tool recently mentioned elsewhere in these comments. I think with the current state of LLMs, harnesses, and related tooling, being able to create or setup self-eval tooling is the biggest differentiator between merely using LLMs to write code vs realizing true 10x productivity wins. I'm curious whether this is something LLMs are eventually going to be good enough at doing, or something the average developer knows when and how to do, or if this is going be something that's too specialized or difficult for most developers and maybe the next generation of developer tooling products. Now that we're several months into Claude Code crossing the threshold of legitimacy and adoption, I've been surprised at how few projects or developers are doing this yet. To a certain extent now all you need to do is ask Claude Code for browser automation workflows and CUJ tests in your repo, and ye shall receive, but probably something that just uses base playwright. It would be even better if you could ask to install or use a self-eval tool that already did everything you needed it to do and also knew how to specify/setup automations. I'm assuming the level of agency or mental overhead of embarking on a browser automation side quest is beyond what most developers are used to in the course of their regular work, even though it's not really as hard as it sounds now. If so then self-eval tooling could be a very promising new product category to sell to enterprises. BTW if you have a link to your project I'd be interested in checking it out! $5.69 for a UAT run sounds very high to me based on how many tokens it typically takes for agents to create automations or steer my similar project, but it could be that your test workloads are much more exhaustive or high-dimensionality than mine are. This is what a basic "go to amazon.com and search for a product, then take a screenshot" automation looks like for ours: https://github.com/accretional/chromerpc/blob/main/recipes/search_and_screenshot.textproto https://github.com/accretional/chromerpc/blob/main/recipes/s.... And this is our interactive/dynamic remote steering mode: https://github.com/accretional/chromerpc/blob/main/chrome-proxy/README.md https://github.com/accretional/chromerpc/blob/main/chrome-pr.... I decided to implement against the Chrome Devtools Protocol (one layer under Playwright) and use grpc service reflection to allow agents to dynamically discover/describe the entire chrome devtools api surface. I just started working on a way to gather traces and monitor/manage the automation run internals because I think there's a ton of opportunity in this problem space for orchestration and RL

by onion2k 4mo ago

alfalfasprout 4mo ago

small downsides like security holes? Those aren't small. Neither is creating a codebase that's an inextensible mess that even LLMs begin to struggle with. The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.

onion2k 4mo ago
small downsides like security holes? Those aren't small I'm British. I've been taught to turn understatement into an art.

hnlmorg 4mo ago

I wouldn’t even trust experienced developers to merge code without peer review.

roenxi 4mo ago
It is getting to the point where I wouldn't trust experienced developers to merge code without AI review. The latest generation of models are getting pretty good.
- IanCal 4mo ago
  Definitely for the time and cost too. If a dev costs $1k/day that’s $30 to spend 15 minutes looking at a pr. How much review does that in tokens get you? I’d wager you’d easily find lots of low level issues that are beyond basic linting with that reliably, and I feel like latest gen models can do really good work.
onion2k 4mo ago
I would, but that's mostly because I don't trust PRs to catch real problems. Someone reviewing the changes in a codebase is never going to spot an architectural or code design issue, and those are the real ones you need to care about. In my experience 95% of everything that's caught by a human PR review could have been caught by a linter or a formatter before the PR was opened. If you trust your team to care about quality then PRs aren't necessary, and if you don't then why are you trusting them to catching problems in PR reviews?
- IanCal 4mo ago
  Outside of the benefit of some extra documentation around changes and having more than one person see what’s happened there are a few main safety parts here: * It can be easier to see a problem in something that another person wrote, you’re not clouded by what you intended to write * different skill sets mean two sets of eyes broadens the kind of problems that can be found (benefits from selecting good reviewers, maybe I’m tagged because I know the llm APIs and performances better by someone who has done refactoring to improve, say, internal caching that’s more their thing) * just chance. There’s some chance you spot an issue, some chance someone else does. Combined it’s better. * I disagree they can’t find architectural or design issues. You see repeated changes of the same kind, or tying together things that shouldn’t be, etc. But yes, many things could be caught before opening. Lots I catch as I’m explaining the change, like rubber ducking. I quite like AI code reviews things for this, there’s a whole back and forth that can be avoided once you get past basic linting/test level things. Save the human time for understanding the higher level issues.

beaugunderson 4mo ago

> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would need to test it myself since MFA requires setup (and a TOTP device!) but no, the agent drove Playwright to change the instance configuration to enable MFA, created a user with MFA enabled, wrote the tiniest (handful of lines) Python TOTP and used that as the device and successfully verified that MFA still worked as expected. I did not expect that ability, and I wrote it! It was also exhaustive in its probe of functionality, way more than I had planned to do, even though I had planned to do a lot given the critical nature of MFA. Total token price for the UAT run: $5.69.

weitendorf 4mo ago
I built a very similar tool recently mentioned elsewhere in these comments. I think with the current state of LLMs, harnesses, and related tooling, being able to create or setup self-eval tooling is the biggest differentiator between merely using LLMs to write code vs realizing true 10x productivity wins. I'm curious whether this is something LLMs are eventually going to be good enough at doing, or something the average developer knows when and how to do, or if this is going be something that's too specialized or difficult for most developers and maybe the next generation of developer tooling products. Now that we're several months into Claude Code crossing the threshold of legitimacy and adoption, I've been surprised at how few projects or developers are doing this yet. To a certain extent now all you need to do is ask Claude Code for browser automation workflows and CUJ tests in your repo, and ye shall receive, but probably something that just uses base playwright. It would be even better if you could ask to install or use a self-eval tool that already did everything you needed it to do and also knew how to specify/setup automations. I'm assuming the level of agency or mental overhead of embarking on a browser automation side quest is beyond what most developers are used to in the course of their regular work, even though it's not really as hard as it sounds now. If so then self-eval tooling could be a very promising new product category to sell to enterprises. BTW if you have a link to your project I'd be interested in checking it out! $5.69 for a UAT run sounds very high to me based on how many tokens it typically takes for agents to create automations or steer my similar project, but it could be that your test workloads are much more exhaustive or high-dimensionality than mine are. This is what a basic "go to amazon.com and search for a product, then take a screenshot" automation looks like for ours: https://github.com/accretional/chromerpc/blob/main/recipes/search_and_screenshot.textproto https://github.com/accretional/chromerpc/blob/main/recipes/s.... And this is our interactive/dynamic remote steering mode: https://github.com/accretional/chromerpc/blob/main/chrome-proxy/README.md https://github.com/accretional/chromerpc/blob/main/chrome-pr.... I decided to implement against the Chrome Devtools Protocol (one layer under Playwright) and use grpc service reflection to allow agents to dynamically discover/describe the entire chrome devtools api surface. I just started working on a way to gather traces and monitor/manage the automation run internals because I think there's a ton of opportunity in this problem space for orchestration and RL