4 ms·

> You wouldn't ask an agent to review a PR then just copy/paste the output int PR would you? Hasn't everyone already got agents directly adding themselves to P

> You wouldn't ask an agent to review a PR then just copy/paste the output int PR would you? Hasn't everyone already got agents directly adding themselves to PRs and leaving comments (occasionally useful)?

nonethewiser 18d ago
They shouldn't. I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code. I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."
- tomrod 18d ago
  Indeed. To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
- alchemism 18d ago
  I think there is a subtle difference between you-or-I dumping a chat paste into the PR, versus an automated agent directly commenting in a PR. The former carries a human social implication, the latter does not. The weight of them is different.
- VBprogrammer 18d ago
  That scenario would be annoying but it's not what we have. The agent joins the PR leaves a few comments and responds (with mixed success) to comments. Personally I'd go back to 2020 and never see an LLM again quite happily but that genie is out of the bottle. Unfortunately, the results speak for themselves and I could probably find dozens of edge cases which the AI code review caught which would have otherwise been missed, probably until production.
MattDamonSpace 18d ago
Yes. I sympathize with the “every PR needs human eyes on it” ethos but fundamentally the only thing that scales with AI PR generation is AI PR review.
- nonethewiser 18d ago
  If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback. To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
- bigstrat2003 18d ago
  That's probably true. And it is also a glaring red flag saying "STOP LETTING AN LLM GENERATE PRs". Far too few will heed that red flag, I fear.
- epolanski 18d ago
  I don't get the point of AI pr review. If the AI can review the pr, it can do so before the PR exists in the first place. It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files. It may actually be even more useful as that person will use it often and improve it or can it.
  - sick_of_slop 18d ago
    [dead]
  - tharkun__ 18d ago
    That sounds good. But it doesn't work in practice from my experience. The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it. And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.

nonethewiser 18d ago

They shouldn't. I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code. I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."

tomrod 18d ago
Indeed. To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
alchemism 18d ago
I think there is a subtle difference between you-or-I dumping a chat paste into the PR, versus an automated agent directly commenting in a PR. The former carries a human social implication, the latter does not. The weight of them is different.
VBprogrammer 18d ago
That scenario would be annoying but it's not what we have. The agent joins the PR leaves a few comments and responds (with mixed success) to comments. Personally I'd go back to 2020 and never see an LLM again quite happily but that genie is out of the bottle. Unfortunately, the results speak for themselves and I could probably find dozens of edge cases which the AI code review caught which would have otherwise been missed, probably until production.

MattDamonSpace 18d ago

Yes. I sympathize with the “every PR needs human eyes on it” ethos but fundamentally the only thing that scales with AI PR generation is AI PR review.

nonethewiser 18d ago
If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback. To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
bigstrat2003 18d ago
That's probably true. And it is also a glaring red flag saying "STOP LETTING AN LLM GENERATE PRs". Far too few will heed that red flag, I fear.
epolanski 18d ago
I don't get the point of AI pr review. If the AI can review the pr, it can do so before the PR exists in the first place. It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files. It may actually be even more useful as that person will use it often and improve it or can it.
- sick_of_slop 18d ago
  [dead]
- tharkun__ 18d ago
  That sounds good. But it doesn't work in practice from my experience. The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it. And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.