5 ms·

Show HN: adamsreview – better multi-agent PR reviews for Claude Code

85 points by adamthegoalie 5mo ago

I built adamsreview, a Claude Code plugin that runs deeper, multi-stage PR reviews using parallel sub-agents, validation passes, persistent JSON state, and optional ensemble review via Codex CLI and PR bot comments. On my own PRs, it has been catching dramatically more real bugs than Claude’s built-in /review, /ultrareview, CodeRabbit, Greptile, and Codex’s built-in review, while producing fewer false positives. adamsreview is six Claude Code slash commands packaged as a plugin: review, codex-review, add, promote, walkthrough, and fix. I modeled it after the built-in /review command and extended it meaningfully. You can clear context between review stages because state is stored in JSON artifacts on disk, with built-in scripts for keeping it updated. The walkthrough command uses Claude’s AskUserQuestion feature to walk you through uncertain findings or items needing human review one by one. Then, the fix command dispatches per-fix-group agents and re-reviews the work with Opus, reverting any regressions before committing survivors. It runs against your regular Claude Code subscription (Max plan recommended), unlike /ultrareview, which charges against your Extra Usage pool. I would love feedback from Claude Code users, pro devs, and anyone with strong opinions about AI code reviews. Repo: https://github.com/adamjgmiller/adamsreview https://github.com/adamjgmiller/adamsreview Install: /plugin marketplace add adamjgmiller/adamsreview, /plugin install adamsreview@adamsreview

esafak 5mo ago
That's looks like a fair bit of ceremony for what it does. Is this representative of the output? https://github.com/adamjgmiller/adamsreview/pull/3 https://github.com/adamjgmiller/adamsreview/pull/3
- adamthegoalie 5mo ago
  No not at all! I haven't used it to review itself actually, for the most part, because adamsreview is mostly english language, not much code. Here's a comment from adamsreview, but even this was 3 weeks ago, and I've worked on it a lot since then: https://github.com/cdinnison/ray-finance/pull/8#issuecomment-4293552360 https://github.com/cdinnison/ray-finance/pull/8#issuecomment... I'll try to find a good public PR to review sometime soon so I can share that and add it to the Readme. This is really good feedback. I should have had something like this ready before posting to Show HN.
- adamthegoalie 5mo ago
  Here's an example: https://github.com/adamjgmiller/worktreehq/pull/145#issuecomment-4423737488 https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...
claudetester89 5mo ago
[dead]
stingraycharles 5mo ago
Holy vibe coding batman this looks like a repository with just a bazillion prompts of which there are already a million. Seems like it would create a lot of friction and burn a lot of tokens.
- adamthegoalie 5mo ago
  It burns a lot of tokens, that's for sure. Friction - maybe? Depending on what you mean. But it's extremely useful and effective compared to everything else out there that I've tried, if you're looking for an AI code review. Let me know if you try it - or find anything else that might work too without the bazillion prompts :)
bilekas 5mo ago
"I pay Claude, to use Claude, to write instructions for Claude, to review code from Claude" Have we all just given up?
- stingraycharles 5mo ago
  You forgot “to use Claude to write a HN post to promote..”
- ramon156 5mo ago
  s/Claude/Intern/g
- naturalintell 5mo ago
  [flagged]
docheinestages 5mo ago
We seem to be fighting complexity with complexity. Does it really help?
- stingraycharles 5mo ago
  This has been the gradual progression of the software world for the past half century, so it’s apt to use LLMs to fight LLMs and call it progress. I wish I was kidding…
  - adamthegoalie 5mo ago
    I totally see where you guys are coming from. It sometimes feels silly to me to have AI reviewing AI reviewing AI all the way down - see my above comment https://news.ycombinator.com/item?id=48095831 https://news.ycombinator.com/item?id=48095831 But with human judgment inserted into the right steps, it's really LLMs leveraging human thought at key stages and then, to your point, LLMs fighting LLMs fighting LLMs all the way down until... Someone like me who has loved software his whole life and never been able to build anything more than a front-end website himself is building entire applications. So maybe it's worth the complexity!
yuppiepuppie 5mo ago
The best code review improvement I have done in my workflow with Claude is using tuicr (https://tuicr.dev https://tuicr.dev). It runs locally, YOU review all the code locally, and feedback that to Claude. Agents reviewing AI code always felt dirty to me, especially when working on production (non-disposable) code.
- ramon156 5mo ago
  The video actually convinced me that this might be an interesting tool. I'm going to try it myself for a small one-shot project and see how well it performs. TUI-based reviews on it's own are already interesting. I had never considered it, I guess.
- free652 5mo ago
  that's a good addition to fresh editor (also tui) and both rust
- mock-possum 5mo ago
  [dead]
- suralind 5mo ago
  I was thinking for years about doing something like this. Thank you for linking this. Would be nice if it allowed to "reject" or mark a change to fix later, but honestly when it would need to be linked to some tracking tool and it would be overkill.
- psadauskas 5mo ago
  I've been using https://github.com/choplin/code-review.nvim https://github.com/choplin/code-review.nvim, which looks like a similar UI, but in the NeoVim interface. `<leader>rc` to comment on a line/selection, then `<leader>ry` to yank all comments into the clipboard to be pasted into a chat. It leaves the comments as markdown files in ./.code-review, so I also have my `/review` agent set to output in the same format, so an LLM can be reviewing the same code I am, I can edit or dismiss the LLM's reviews, then send the whole thing back to the first agent to fix.
- 5mo ago
thesimon 5mo ago
> Runs against your regular Claude Code subscription (Max plan recommended) — unlike /ultrareview, which charges against your Extra Usage pool. How expensive is it to run in your experience? In $ or tokens?
- adamthegoalie 5mo ago
  On small PRs (small features / changes ~hundreds to thousands of lines), I'd say around 500,000 total tokens. On large PRs (new feature sets for apps ~10,000-30,000 lines), around 2-3 million total tokens. By the way, I should have mentioned in my original post, adamsreview counts tokens used by sub-agents across the stages, and tells you at the end of each stage the total used so far.
  - dayjaby 5mo ago
    Since when are thousands of lines of code changes "a small PR"?
    - adamthegoalie 5mo ago
      Fair point. But since Claude Code I guess? :D
Jinyibruceli 5mo ago
[flagged]
nkmnz 5mo ago
Great project! I’ve build something similar, not very clean and polished, but focussed around deterministic orchestration of multiple agents via typescript, because a coordinating agent was notoriously bad at things such as fetching relevant tickets and other context. One thing I struggle with so far, though, are the actual instructions for the review themselves. They are either too vague, leading to superficial or overly broad reviews, or too specific and thus not applicable to different kinds of PRs…
- adamthegoalie 5mo ago
  That's awesome to hear and I'd love to see it when you're ready. I actually think having something like adamsreview orchestrated by deterministic code - instead of simply having AI agents use deterministic code occasionally as this app does - could be even better! The problem I ran into is that if you build a deterministic app that happens to use LLMs instead of the other way around, I don't think there's any way to get it to use your Claude Code subscription credits. It has to use API. And something like adamsreview would end up being so expensive if not subsidized by Anthropic along with the rest of our CC usage. Curious to hear about your experience.
  - nkmnz 5mo ago
    It's possible to use subscriptions! I run them in containers. For claude, I use `claude setup-token`, put the token into a local auth.json and mount that. For codex, I run the cli in my working dir prefixed with `CODEX_HOME=./codex-home codex` and mount that whole `codex-home` directory - done :)
- azurewraith 5mo ago
  You've touched on the core issue... prompt based orchestration is non-idempotent. Same input, different orchestration paths every run. I've had better experience enforcing tool restrictions in discrete phases with deterministic code over orchestrating via LLM. Prompt level instructions get ignored, it's much harder to ignore a protocol-level tool restriction. It's simple and yet oddly VERY effective.
Ozzie-D 5mo ago
[flagged]
- adamthegoalie 5mo ago
  Do you mean that, even though the tooling keeps getting better, people aren't putting the effort into using it?
moomin 5mo ago
Is there a good way of adding in your own rules to the review? I’m always in the market for better review tools but I also need to check against internal coding stands and expectations,
- adamthegoalie 5mo ago
  Definitely! adamsreview is mostly english language - since it is instructing your CC agents. While there are a number of python scripts and JSON storage, you - and your agent - would easily be able to add your own rules to this. It also respects your Claude.md and one of the lenses it already uses is checking for Claude.md compliance etc.
parasmadan 5mo ago
Why not just use an eval harness to prove this catches more real bugs? Benchmarks on actual bug classes would be far more convincing than comparing against /review.
- stingraycharles 5mo ago
  That’s probably more work than the entire repo itself. Would need to be something like SWE-bench with and without “adamsreview”. You’re right though, but evals are actually fairly tricky to write and maintain.
- adamthegoalie 5mo ago
  That's a great idea. I had trouble finding anything like this, a benchmark made for (AI) code reviewers. I had expected to find something like an eval harness available on GitHub, but couldn't find it. Any suggestions? Or maybe we/I/someone should build something like this? I suppose one challenge is that if it's going to be publicly available, it would also be easy to cheat, but still seems it would be useful if people agreed it's a good benchmark and could easily re-test tools themselves.
  - phonon 5mo ago
    https://www.codereviewbench.com/ https://www.codereviewbench.com/ https://codereview.withmartian.com/ https://codereview.withmartian.com/
    - wordpad 5mo ago
      Based on how same models rank fluctuates week to week, all I can conclude is that no frontier models is statistically better than the other or it's too task dependent that the result cannot converge.
    - adamthegoalie 5mo ago
      I will be trying these and will report back!
  - 5mo ago
ltononro 5mo ago
[dead]
momentmaker 5mo ago
[dead]
alex1sa 5mo ago
[dead]
madamelic 5mo ago
Neat idea. I am more curious about your AI workflow as I stay away from other's tools because I don't trust vibe-code related tools. What is the workflow difference between `fragments/` and `plans/`. They seem logically the same but seem to have been used for different purposes. Is this something it did on its own or is this something you prompted it to do?
- adamthegoalie 5mo ago
  Hey thanks for the comment and the question. I would say my workflow for any meaningful amount of work is (all in Claude Code): - PRD: I discuss and brainstorm with Claude Code using something like the Grill Me skill https://github.com/mattpocock/skills/tree/main/skills/productivity/grill-me https://github.com/mattpocock/skills/tree/main/skills/produc... but that I've modified a bit for my own style, until I have a good PRD (what the goals / design decisions are for what I'm building) --- I run this PRD through multiple AI reviews (sometimes ChatGPT Pro for really important PRDs, because it seems to have some of the best critical feedback) --- I read the PRD myself in detail before finalizing. - PLAN: I have Claude Code develop the plan for implementing the PRD. Again, I have this reviewed several times by CC and sometimes by other tools for effectiveness, consistency with the PRD, consistency with the codebase, and internal consisenty. - EXECUTE: I have an orchestration command I made that has CC execute the PLAN and use a build journal, using sub-agents whenever possible to save context, so that it can operate for up to several hours. - QUICK REVIEWS: I have these commands /review-fix-loop /quick-dual-review which loops around running a Claude+Codex sub agents review and then fixing anything critical (deferring items needing human judgment) - CODE REVIEWS: This is when I run between one and several of the adamsreview reviews, starting with /review --ensemble, then /walkthrough, then /fix; until I am satisfied. Would it be useful if I packaged all this stuff into a GH repo to share with you and others?
  - esperent 5mo ago
    I think there's no harm in that. But I will say, it sounds very similar to my own system, and probably a ton of other people's. Yours might very well be better than most, but the thing that's missing from all of these is evals. You, me, everyone else, we're all vibe coding up these loops, we're getting work done and feeling excited about it, excited enough that we want to share, but nobody is doing real testing or benchmarks.
volume_tech 5mo ago
[flagged]
kuri-sun 5mo ago
Curious what kinds of bugs the multi-agent setup catches thatsingle-pass review misses in practice. Is it more about coverage(different agents looking at different aspects) or about getting a second opinion on the same aspect? The README has examples but the mechanism by which the parallelism actually helps isn't obvious to me from them.
- lmeyerov 5mo ago
  Yes, being comprehensive, so early or blatant cheapo findings do not distract from other ones. That's important for base results. Splitting in both file and task is (currently) important. Additionally, we run in a loop until it stops finding things, and as part of that, do test amplification when it does find any. We regularly see 3-8 rounds yielding valid results. IMO half the value is customization to your repo, so copying these and specializing to your repo is super quick and pays off almost immediately . How to find style guides, how to run tests, what dimensions of correctness to look for, etc. We do a similar look here: https://github.com/graphistry/pygraphistry/blob/master/agents/skills/review/SKILL.md https://github.com/graphistry/pygraphistry/blob/master/agent... This kind of thing makes me question how important Mythos is for security bug finding - doing a High effort loop with a frontier model in code reviews until convergence has already outperformed human review for us . (Doesn't replace, but does find things we miss, and catches many we do see earlier).
  - esperent 5mo ago
    How do you prevent it from increasing scope? That's the main issue I've found from running loops like this. Each loop has ~7 agents, say, looking through different lenses (security, UX, performance, etc.). Each one notes a few issues, each issue gets fixed, you do 5 to 8 loops, as you say. Each individual item that gets fixed looks minor but when you add it all up at the end you've increased PR size and scope significantly.
    - adamthegoalie 5mo ago
      That is such a good point. I recently opened a PR against this AI personal finance tool Ray https://github.com/cdinnison/ray-finance/pull/8 https://github.com/cdinnison/ray-finance/pull/8 to add an Apple Card import feature, since Apple Card is not supported by Plaid. I built the manual import feature, opened the PR, and then ran a code review. What I hadn't thought about when I built the feature, was the myriad ways that the implications of importing data from Apple would have to be considered and integrated into the rest of the app, for the manual import to be a first-class feature, not "just a manual import" of data. I ended up running adamsreview against it like 5-10 times, before considering it complete, as I learned that there was much more to the integration than I realized. Now is that necessarily a problem? Maybe not. I should have realized from the start that the import feature was going to much more than just a small feature. But at least, thanks to the review loop, I got it completely right before the PR was merged.
adamthegoalie 5mo ago
Does anyone have an open PR on a public repo? I'll run this against your PR for you with my CC credits as a sort-of benchmark! Send me your PR link :) I'm going to create one on one of my other repos meanwhile and add a link to the review when it's ready.
- adamthegoalie 5mo ago
  Meanwhile here's what it looks like - just ran it on my own other repo: https://github.com/adamjgmiller/worktreehq/pull/145#issuecomment-4423737488 https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...
- michaelsalim 5mo ago
  Quite a huge one, but here's one :) https://github.com/Vija02/TheOpenPresenter/pull/170 https://github.com/Vija02/TheOpenPresenter/pull/170
  - adamthegoalie 5mo ago
    Yesss! On it now! 1-2 hours you’ll get a comment
  - adamthegoalie 5mo ago
    Your review is ready sir https://github.com/Vija02/TheOpenPresenter/pull/170#issuecomment-4435243615 https://github.com/Vija02/TheOpenPresenter/pull/170#issuecom... This was claude-only, no codex / --ensemble mode. It ran against the branch's base, which is 1 behind main. Would love your feedback when you have the chance!
deleted 5mo ago
[deleted]
quantumadopter 5mo ago
[flagged]
- adamthegoalie 5mo ago
  Well, it doesn’t give you an opinion on whether you should merge or not. It gives you a list of issues along with details about those issues, such as fix hints, and whether you need human attention before fixing. It’s all given to you in a structured file, in your chat, and as a nicely formatted PR comment. Still up to the human to decide whether and when to merge based on the output!
  - quantumadopter 5mo ago
    [flagged]
ki_sum_ai 5mo ago
[dead]
bmarchal 5mo ago
[flagged]
kcarriedo 5mo ago
[flagged]

85 points by adamthegoalie 5mo ago

esafak 5mo ago

That's looks like a fair bit of ceremony for what it does. Is this representative of the output? https://github.com/adamjgmiller/adamsreview/pull/3 https://github.com/adamjgmiller/adamsreview/pull/3

adamthegoalie 5mo ago
No not at all! I haven't used it to review itself actually, for the most part, because adamsreview is mostly english language, not much code. Here's a comment from adamsreview, but even this was 3 weeks ago, and I've worked on it a lot since then: https://github.com/cdinnison/ray-finance/pull/8#issuecomment-4293552360 https://github.com/cdinnison/ray-finance/pull/8#issuecomment... I'll try to find a good public PR to review sometime soon so I can share that and add it to the Readme. This is really good feedback. I should have had something like this ready before posting to Show HN.
adamthegoalie 5mo ago
Here's an example: https://github.com/adamjgmiller/worktreehq/pull/145#issuecomment-4423737488 https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...

claudetester89 5mo ago

[dead]

stingraycharles 5mo ago

Holy vibe coding batman this looks like a repository with just a bazillion prompts of which there are already a million. Seems like it would create a lot of friction and burn a lot of tokens.

adamthegoalie 5mo ago
It burns a lot of tokens, that's for sure. Friction - maybe? Depending on what you mean. But it's extremely useful and effective compared to everything else out there that I've tried, if you're looking for an AI code review. Let me know if you try it - or find anything else that might work too without the bazillion prompts :)

bilekas 5mo ago

"I pay Claude, to use Claude, to write instructions for Claude, to review code from Claude" Have we all just given up?

stingraycharles 5mo ago
You forgot “to use Claude to write a HN post to promote..”
ramon156 5mo ago
s/Claude/Intern/g
naturalintell 5mo ago
[flagged]

docheinestages 5mo ago

We seem to be fighting complexity with complexity. Does it really help?

stingraycharles 5mo ago
This has been the gradual progression of the software world for the past half century, so it’s apt to use LLMs to fight LLMs and call it progress. I wish I was kidding…
- adamthegoalie 5mo ago
  I totally see where you guys are coming from. It sometimes feels silly to me to have AI reviewing AI reviewing AI all the way down - see my above comment https://news.ycombinator.com/item?id=48095831 https://news.ycombinator.com/item?id=48095831 But with human judgment inserted into the right steps, it's really LLMs leveraging human thought at key stages and then, to your point, LLMs fighting LLMs fighting LLMs all the way down until... Someone like me who has loved software his whole life and never been able to build anything more than a front-end website himself is building entire applications. So maybe it's worth the complexity!

yuppiepuppie 5mo ago

The best code review improvement I have done in my workflow with Claude is using tuicr (https://tuicr.dev https://tuicr.dev). It runs locally, YOU review all the code locally, and feedback that to Claude. Agents reviewing AI code always felt dirty to me, especially when working on production (non-disposable) code.

ramon156 5mo ago
The video actually convinced me that this might be an interesting tool. I'm going to try it myself for a small one-shot project and see how well it performs. TUI-based reviews on it's own are already interesting. I had never considered it, I guess.
free652 5mo ago
that's a good addition to fresh editor (also tui) and both rust
mock-possum 5mo ago
[dead]
suralind 5mo ago
I was thinking for years about doing something like this. Thank you for linking this. Would be nice if it allowed to "reject" or mark a change to fix later, but honestly when it would need to be linked to some tracking tool and it would be overkill.
psadauskas 5mo ago
I've been using https://github.com/choplin/code-review.nvim https://github.com/choplin/code-review.nvim, which looks like a similar UI, but in the NeoVim interface. `<leader>rc` to comment on a line/selection, then `<leader>ry` to yank all comments into the clipboard to be pasted into a chat. It leaves the comments as markdown files in ./.code-review, so I also have my `/review` agent set to output in the same format, so an LLM can be reviewing the same code I am, I can edit or dismiss the LLM's reviews, then send the whole thing back to the first agent to fix.
5mo ago

thesimon 5mo ago

> Runs against your regular Claude Code subscription (Max plan recommended) — unlike /ultrareview, which charges against your Extra Usage pool. How expensive is it to run in your experience? In $ or tokens?

adamthegoalie 5mo ago
On small PRs (small features / changes ~hundreds to thousands of lines), I'd say around 500,000 total tokens. On large PRs (new feature sets for apps ~10,000-30,000 lines), around 2-3 million total tokens. By the way, I should have mentioned in my original post, adamsreview counts tokens used by sub-agents across the stages, and tells you at the end of each stage the total used so far.
- dayjaby 5mo ago
  Since when are thousands of lines of code changes "a small PR"?
  - adamthegoalie 5mo ago
    Fair point. But since Claude Code I guess? :D

Jinyibruceli 5mo ago

[flagged]

nkmnz 5mo ago

Great project! I’ve build something similar, not very clean and polished, but focussed around deterministic orchestration of multiple agents via typescript, because a coordinating agent was notoriously bad at things such as fetching relevant tickets and other context. One thing I struggle with so far, though, are the actual instructions for the review themselves. They are either too vague, leading to superficial or overly broad reviews, or too specific and thus not applicable to different kinds of PRs…

adamthegoalie 5mo ago
That's awesome to hear and I'd love to see it when you're ready. I actually think having something like adamsreview orchestrated by deterministic code - instead of simply having AI agents use deterministic code occasionally as this app does - could be even better! The problem I ran into is that if you build a deterministic app that happens to use LLMs instead of the other way around, I don't think there's any way to get it to use your Claude Code subscription credits. It has to use API. And something like adamsreview would end up being so expensive if not subsidized by Anthropic along with the rest of our CC usage. Curious to hear about your experience.
- nkmnz 5mo ago
  It's possible to use subscriptions! I run them in containers. For claude, I use `claude setup-token`, put the token into a local auth.json and mount that. For codex, I run the cli in my working dir prefixed with `CODEX_HOME=./codex-home codex` and mount that whole `codex-home` directory - done :)
azurewraith 5mo ago
You've touched on the core issue... prompt based orchestration is non-idempotent. Same input, different orchestration paths every run. I've had better experience enforcing tool restrictions in discrete phases with deterministic code over orchestrating via LLM. Prompt level instructions get ignored, it's much harder to ignore a protocol-level tool restriction. It's simple and yet oddly VERY effective.

Ozzie-D 5mo ago

[flagged]

adamthegoalie 5mo ago
Do you mean that, even though the tooling keeps getting better, people aren't putting the effort into using it?

moomin 5mo ago

Is there a good way of adding in your own rules to the review? I’m always in the market for better review tools but I also need to check against internal coding stands and expectations,

adamthegoalie 5mo ago
Definitely! adamsreview is mostly english language - since it is instructing your CC agents. While there are a number of python scripts and JSON storage, you - and your agent - would easily be able to add your own rules to this. It also respects your Claude.md and one of the lenses it already uses is checking for Claude.md compliance etc.

parasmadan 5mo ago

Why not just use an eval harness to prove this catches more real bugs? Benchmarks on actual bug classes would be far more convincing than comparing against /review.

stingraycharles 5mo ago
That’s probably more work than the entire repo itself. Would need to be something like SWE-bench with and without “adamsreview”. You’re right though, but evals are actually fairly tricky to write and maintain.
adamthegoalie 5mo ago
That's a great idea. I had trouble finding anything like this, a benchmark made for (AI) code reviewers. I had expected to find something like an eval harness available on GitHub, but couldn't find it. Any suggestions? Or maybe we/I/someone should build something like this? I suppose one challenge is that if it's going to be publicly available, it would also be easy to cheat, but still seems it would be useful if people agreed it's a good benchmark and could easily re-test tools themselves.
- phonon 5mo ago
  https://www.codereviewbench.com/ https://www.codereviewbench.com/ https://codereview.withmartian.com/ https://codereview.withmartian.com/
  - wordpad 5mo ago
    Based on how same models rank fluctuates week to week, all I can conclude is that no frontier models is statistically better than the other or it's too task dependent that the result cannot converge.
  - adamthegoalie 5mo ago
    I will be trying these and will report back!
- 5mo ago

ltononro 5mo ago

[dead]

momentmaker 5mo ago

[dead]

alex1sa 5mo ago

[dead]

madamelic 5mo ago

Neat idea. I am more curious about your AI workflow as I stay away from other's tools because I don't trust vibe-code related tools. What is the workflow difference between `fragments/` and `plans/`. They seem logically the same but seem to have been used for different purposes. Is this something it did on its own or is this something you prompted it to do?

adamthegoalie 5mo ago
Hey thanks for the comment and the question. I would say my workflow for any meaningful amount of work is (all in Claude Code): - PRD: I discuss and brainstorm with Claude Code using something like the Grill Me skill https://github.com/mattpocock/skills/tree/main/skills/productivity/grill-me https://github.com/mattpocock/skills/tree/main/skills/produc... but that I've modified a bit for my own style, until I have a good PRD (what the goals / design decisions are for what I'm building) --- I run this PRD through multiple AI reviews (sometimes ChatGPT Pro for really important PRDs, because it seems to have some of the best critical feedback) --- I read the PRD myself in detail before finalizing. - PLAN: I have Claude Code develop the plan for implementing the PRD. Again, I have this reviewed several times by CC and sometimes by other tools for effectiveness, consistency with the PRD, consistency with the codebase, and internal consisenty. - EXECUTE: I have an orchestration command I made that has CC execute the PLAN and use a build journal, using sub-agents whenever possible to save context, so that it can operate for up to several hours. - QUICK REVIEWS: I have these commands /review-fix-loop /quick-dual-review which loops around running a Claude+Codex sub agents review and then fixing anything critical (deferring items needing human judgment) - CODE REVIEWS: This is when I run between one and several of the adamsreview reviews, starting with /review --ensemble, then /walkthrough, then /fix; until I am satisfied. Would it be useful if I packaged all this stuff into a GH repo to share with you and others?
- esperent 5mo ago
  I think there's no harm in that. But I will say, it sounds very similar to my own system, and probably a ton of other people's. Yours might very well be better than most, but the thing that's missing from all of these is evals. You, me, everyone else, we're all vibe coding up these loops, we're getting work done and feeling excited about it, excited enough that we want to share, but nobody is doing real testing or benchmarks.

volume_tech 5mo ago

[flagged]

kuri-sun 5mo ago

Curious what kinds of bugs the multi-agent setup catches thatsingle-pass review misses in practice. Is it more about coverage(different agents looking at different aspects) or about getting a second opinion on the same aspect? The README has examples but the mechanism by which the parallelism actually helps isn't obvious to me from them.

lmeyerov 5mo ago
Yes, being comprehensive, so early or blatant cheapo findings do not distract from other ones. That's important for base results. Splitting in both file and task is (currently) important. Additionally, we run in a loop until it stops finding things, and as part of that, do test amplification when it does find any. We regularly see 3-8 rounds yielding valid results. IMO half the value is customization to your repo, so copying these and specializing to your repo is super quick and pays off almost immediately . How to find style guides, how to run tests, what dimensions of correctness to look for, etc. We do a similar look here: https://github.com/graphistry/pygraphistry/blob/master/agents/skills/review/SKILL.md https://github.com/graphistry/pygraphistry/blob/master/agent... This kind of thing makes me question how important Mythos is for security bug finding - doing a High effort loop with a frontier model in code reviews until convergence has already outperformed human review for us . (Doesn't replace, but does find things we miss, and catches many we do see earlier).
- esperent 5mo ago
  How do you prevent it from increasing scope? That's the main issue I've found from running loops like this. Each loop has ~7 agents, say, looking through different lenses (security, UX, performance, etc.). Each one notes a few issues, each issue gets fixed, you do 5 to 8 loops, as you say. Each individual item that gets fixed looks minor but when you add it all up at the end you've increased PR size and scope significantly.
  - adamthegoalie 5mo ago
    That is such a good point. I recently opened a PR against this AI personal finance tool Ray https://github.com/cdinnison/ray-finance/pull/8 https://github.com/cdinnison/ray-finance/pull/8 to add an Apple Card import feature, since Apple Card is not supported by Plaid. I built the manual import feature, opened the PR, and then ran a code review. What I hadn't thought about when I built the feature, was the myriad ways that the implications of importing data from Apple would have to be considered and integrated into the rest of the app, for the manual import to be a first-class feature, not "just a manual import" of data. I ended up running adamsreview against it like 5-10 times, before considering it complete, as I learned that there was much more to the integration than I realized. Now is that necessarily a problem? Maybe not. I should have realized from the start that the import feature was going to much more than just a small feature. But at least, thanks to the review loop, I got it completely right before the PR was merged.

adamthegoalie 5mo ago

Does anyone have an open PR on a public repo? I'll run this against your PR for you with my CC credits as a sort-of benchmark! Send me your PR link :) I'm going to create one on one of my other repos meanwhile and add a link to the review when it's ready.

adamthegoalie 5mo ago
Meanwhile here's what it looks like - just ran it on my own other repo: https://github.com/adamjgmiller/worktreehq/pull/145#issuecomment-4423737488 https://github.com/adamjgmiller/worktreehq/pull/145#issuecom...
michaelsalim 5mo ago
Quite a huge one, but here's one :) https://github.com/Vija02/TheOpenPresenter/pull/170 https://github.com/Vija02/TheOpenPresenter/pull/170
- adamthegoalie 5mo ago
  Yesss! On it now! 1-2 hours you’ll get a comment
- adamthegoalie 5mo ago
  Your review is ready sir https://github.com/Vija02/TheOpenPresenter/pull/170#issuecomment-4435243615 https://github.com/Vija02/TheOpenPresenter/pull/170#issuecom... This was claude-only, no codex / --ensemble mode. It ran against the branch's base, which is 1 behind main. Would love your feedback when you have the chance!

deleted 5mo ago

[deleted]

quantumadopter 5mo ago

[flagged]

adamthegoalie 5mo ago
Well, it doesn’t give you an opinion on whether you should merge or not. It gives you a list of issues along with details about those issues, such as fix hints, and whether you need human attention before fixing. It’s all given to you in a structured file, in your chat, and as a nicely formatted PR comment. Still up to the human to decide whether and when to merge based on the output!
- quantumadopter 5mo ago
  [flagged]

ki_sum_ai 5mo ago

[dead]

bmarchal 5mo ago

[flagged]

kcarriedo 5mo ago

[flagged]