3 ms·

Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that in

Easy, have another agent check it. Yeah, I know, just more slop. But I do think the second agent’s eagerness to please is aligned more in your favor in that instance, so it’s likely to find most issues. The bigger problem I’ve found is that it’ll also find all kinds of very minor edge cases that you have to pick through.

Forgeties79 2mo ago
I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase.
- briHass 2mo ago
  The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it.
  - Forgeties79 2mo ago
    It just seems like everybody I know is realizing they’ve built way too much to possibly wade through and a lot of it is completely unnecessary. Using the tool that created the problem to solve the problem does not seem like the best course of action to me.
- agumonkey 2mo ago
  Management still pushes for more ai and will rather hire more heads to "handle" issues.
  - Forgeties79 2mo ago
    Hiring? Seems to me that market’s rough right now and AI is being used for cost cutting.
    - agumonkey 2mo ago
      I know but in our case they really did put out a few job ads. The market is full of different small contexts where things are a bit flipped.
      - Forgeties79 2mo ago
        I’m glad to hear it truly
        agumonkey 2mo ago
        Yeah I guess the job market will not disappear as fast as I could believe when Claude code came out. Maybe people can enjoy a few more years of work.
noir_lord 2mo ago
Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?"
- iterateoften 2mo ago
  I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents
- senordevnyc 2mo ago
  YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes.
- Qworg 2mo ago
  Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge.
skybrian 2mo ago
The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked.
hirako2000 2mo ago
Then you end up with the false positives. I once made a counter judge, and a loop to make corrections deemed true positives. The loop cost me a lot and still left the results to be desirable.
nprateem 2mo ago
Two things worth flagging: [Claude proceeds to waste your time telling you about bugs it caused then fixed and other non-issues...] Really wish they'd get rid of this. It must be in the system prompt as it always 'flags' 2 things

Forgeties79 2mo ago

I do not understand how some of y’all are not under water with fragile code that is too massive to possibly parse. Every engineering team I know is currently trying to undo the damage of the last 6-12mo when they all got more serious into adopting these tools (usually Claude). It hasn’t completely screwed them over, but the the debt is substantial and cannot be put off anymore it seems. They argue the net is positive but clearly the “100x productivity multiplier” claims have been dashed on the shoals of reality for these groups. This is anecdotal, but it’s across the board in my vicinity. I’m curious how common this is and if it’s just “the new normal” to adopt the nauseating Covid phrase.

briHass 2mo ago
The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards. These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals. They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it.
- Forgeties79 2mo ago
  It just seems like everybody I know is realizing they’ve built way too much to possibly wade through and a lot of it is completely unnecessary. Using the tool that created the problem to solve the problem does not seem like the best course of action to me.
agumonkey 2mo ago
Management still pushes for more ai and will rather hire more heads to "handle" issues.
- Forgeties79 2mo ago
  Hiring? Seems to me that market’s rough right now and AI is being used for cost cutting.
  - agumonkey 2mo ago
    I know but in our case they really did put out a few job ads. The market is full of different small contexts where things are a bit flipped.
    - Forgeties79 2mo ago
      I’m glad to hear it truly
      - agumonkey 2mo ago
        Yeah I guess the job market will not disappear as fast as I could believe when Claude code came out. Maybe people can enjoy a few more years of work.

noir_lord 2mo ago

Do we add a third one to check the second one which is checking the first? Asking slightly tongue in cheek but at what point does this stop making sense if we can't trust the output, the people creating the models are already getting surprised in bad ways (if we take their words at face value) with how the models are behaving already etc. We have the folks over here saying "AI is amazing" and the other other folks over there saying "AI is terrible". I've largely sat it out so far and I listen to both camps (and people in the middle as well) and I keep half an eye on what they are up to (including periodically evaluating them) but my overarching impression is still "Why would we trust this when it hasn't shown it's trustworthy?"

iterateoften 2mo ago
I mean if each agent reduces probability of error by 90% then after 9 agents you would have “nine nines” of reliability. Obviously maybe it’s not composable like that exactly in real world but that’s the intent of agents checking agents
senordevnyc 2mo ago
YMMV, I’m pretty AI-pilled in the sense that I think AI is one of the most pivotal things humanity will ever invent, and it’s going to radically change our civilization over the next few decades (not necessarily for the better!) But I wouldn’t say I “trust” these agents. The degree to which I double check their work depends heavily on the consequences if it gets something wrong. Not too dissimilar from another human dev in that sense. So for the SaaS that supports my family, there are some things I have it build where I glance at the PR for a minute or two, but if it broke something on this admin page that only I see, there’s no real downside and I’ll find out pretty quickly next time I use it. And it’s fine 95% of the time, so it doesn’t feel like the best use of my time to double-check it carefully. But for some of the complex internal flows where a bug could be both catastrophic and difficult to even discover for awhile, I still check it very carefully. For a little one-off vibe coded demo thing like OP shared, I wouldn’t look at the code at all, I’d just have another agent check it and fix anything it finds. Very low stakes.
Qworg 2mo ago
Humans are also not trustworthy. So we build systems to capture their nondeterministic output and solve the issues that emerge.

skybrian 2mo ago

The "second" agent could also be the same one with a different prompt. LLMs aren't attached to their previous output; they'll point out problems if asked.

hirako2000 2mo ago

Then you end up with the false positives. I once made a counter judge, and a loop to make corrections deemed true positives. The loop cost me a lot and still left the results to be desirable.

nprateem 2mo ago

Two things worth flagging: [Claude proceeds to waste your time telling you about bugs it caused then fixed and other non-issues...] Really wish they'd get rid of this. It must be in the system prompt as it always 'flags' 2 things