3 ms·

Our team has pretty aggressively started using LLMs for automated code review. It will look at our PRs and post comments. We can adding more material for differ

Our team has pretty aggressively started using LLMs for automated code review. It will look at our PRs and post comments. We can adding more material for different things for it to consider- from a looking at a summarized version of our API guidelines, general prompts like, "You are an expert software engineer and QA professional, review this PR and point out any bugs or other areas of technical risk. Make concise suggestions for improvement where applicable." - it catches a ton of stuff. Another area we've started doing is having it look at build failures and writing a report on suggested root causes before even a human looks at it - saves time. Or (and we haven't rolled this out automatically yet but are testing a prototype) having it triage alarms from our metrics, with access to the logs and codebase to investigate.

infecto 1y ago
I have been surprised more folks have no rolled these out as paid for products. I have been getting tons of use out of systems like cursors bugbot. The signal to noise is high and while it’s not always right it catches a lot of bugs I would have missed.
- senko 1y ago
  There are a few: Greptile, Ellipsis, GH Copilot (integrated with GH) I feel many also try "review and fix automatically", as it's tempting to "just" pass the generated comments to a second agent to apply them. But that opens a whole other can of worms and pretty soon you're just another code assistant service.
  - dearilos 1y ago
    if you do specific prompts based on your team's tribal knowledge and standards it works really well "look at this code for bugs" doesn't end up working well, which is what most code reviewers do.
    - jacobr1 1y ago
      Yep - this is the key to making it work
dearilos 1y ago
how are you dealing with the accuracy of the review comments? in my experience "review this PR" is very generic and ends up giving slop.
- jacobr1 1y ago
  Yeah - that doesn't work. We include our own guidelines and practices in the context. Basically an ever evolving wiki page of PR best practices. We had that before trying LLMs - so it was easier for us to start. Also we found doing an LLM reformat of that data to might a much tighter set of "rules" helped as well.
  - dearilos 1y ago
    ive been building out a directory of code review rules for the last couple of months! are you open to chatting and sharing notes on what works/doesn't work? my email is ilya (at) wispbit.com

infecto 1y ago

I have been surprised more folks have no rolled these out as paid for products. I have been getting tons of use out of systems like cursors bugbot. The signal to noise is high and while it’s not always right it catches a lot of bugs I would have missed.

senko 1y ago
There are a few: Greptile, Ellipsis, GH Copilot (integrated with GH) I feel many also try "review and fix automatically", as it's tempting to "just" pass the generated comments to a second agent to apply them. But that opens a whole other can of worms and pretty soon you're just another code assistant service.
- dearilos 1y ago
  if you do specific prompts based on your team's tribal knowledge and standards it works really well "look at this code for bugs" doesn't end up working well, which is what most code reviewers do.
  - jacobr1 1y ago
    Yep - this is the key to making it work

dearilos 1y ago

how are you dealing with the accuracy of the review comments? in my experience "review this PR" is very generic and ends up giving slop.

jacobr1 1y ago
Yeah - that doesn't work. We include our own guidelines and practices in the context. Basically an ever evolving wiki page of PR best practices. We had that before trying LLMs - so it was easier for us to start. Also we found doing an LLM reformat of that data to might a much tighter set of "rules" helped as well.
- dearilos 1y ago
  ive been building out a directory of code review rules for the last couple of months! are you open to chatting and sharing notes on what works/doesn't work? my email is ilya (at) wispbit.com