4 ms·

I'm in the market for PR review bots, as the nitpicking issue is real. So far, I have tried Coderabbit, but it adds too much noise to PRs, and only a very small

I'm in the market for PR review bots, as the nitpicking issue is real. So far, I have tried Coderabbit, but it adds too much noise to PRs, and only a very small percentage of comments are actually useful. I specifically instructed it to ignore nitpicks, but it still added such comments. Their cringy ASCII art comments make it even harder to take them seriously. I recently signed up for Korbit AI, but it's too soon to provide feedback. Honestly, I’m getting a bit fed up with experimenting with different PR bots. Question for the author: In what ways is your solution better than Coderabbit and Korbit AI?

dakshgupta 2y ago
I haven’t explored Korbit but with CodeRabbit there are a couple of things: 1. We are better at full codebase context, because of how we index the codebase like a graph and use graph search and an LLM to determine what other parts of the codebase should be taken into consideration while reviewing a diff. 2. We offer an API you can use to build custom workflows, for example every time a test fails in your pipeline you can pass the output to Greptile and it will diagnose with full codebase context. 3. Customers of ours that switched from CodeRabbit usually say Greptile has far fewer and less verbose comments. This is a subjective, of course. That said, CodeRabbit is cheaper. Both products have free trials for though, so I would recommend trying both and seeing which one your team prefers, which is ultimately what matters. I’m happy to answer more questions or a demo for your team too -> daksh@greptile.com
swells34 2y ago
Isn't code review the one area you'd never want to replace the programmer in? Like, that is the most important step of the process; it's where we break down the logic and sanity check the implementation. That is expressly where AI is the weakest.
- imoverclocked 2y ago
  Many code review products are not AI. They are often collections of rules that have been specifically crafted to catch bad patterns. Some of the rules are stylistic in nature and those tend to turn people away the fastest IMHO. Many of these tools can be integrated into a local workflow so they will never ping you on a PR but many developers prefer to just write some code and let the review process suss things out.
- yorwba 2y ago
  You probably shouldn't take a code review bot's "LGTM!" at face value, but if it tells you there is an issue with your code and it is indeed an issue with your code, it has successfully subbed in for your coworker who didn't get around to looking at it yet.
  - imiric 2y ago
    > if it tells you there is an issue with your code and it is indeed an issue with your code That's the crucial bit. If it ends up hallucinating issues as LLMs tend to do[1], then it just adds more work for human developers to confirm its report. Human devs can create false reports as well, but we're more likely to miss an issue than misunderstand and be confident about it. [1]: https://www.theregister.com/2024/12/10/ai_slop_bug_reports/ https://www.theregister.com/2024/12/10/ai_slop_bug_reports/
- dakshgupta 2y ago
  I agree - at least with where technology is today, I would strongly discourage replacing human code review with AI. It does however serve as a good first pass, so by the time the human reviewer gets it, the little things have been addressed and they can focus on broader technical decisions. Would you want your coworkers to run their PRs through an AI reviewer, resolve those comments, then send it to you?
  - deleted 2y ago
    [deleted]
  - AgentOrange1234 2y ago
    This feels like the right take. Once we have an AI starting to do something, there’s an art to gradually adopting it in a way that makes sense. We don’t lose the ability to have humans review code. We don’t have to all use this on every PR. We don’t have to blindly accept every comment it makes.
- jerrygoyal 2y ago
  No one is blindly merging PR after AI Code Review. Think of review bot as an extra pair of eyes.
  - bobnamob 2y ago
    Yet. The things I've seen juniors try to do because "the AI said so" is staggering. Furthermore, I have little faith that the average junior gets competent enough teaching/mentoring that this attitude won't be a pervasive problem in 5ish years. Admittedly, maybe I'm getting old and this is just another "darn kids these days" rant
- hansvm 2y ago
  If you think of the "AI" benefit as more of a linter then the value starts to become clear. E.g., I've never hesitated to add a linting rule requiring `except Exception:` in lieu of `except:` in Python, since the latter is very rarely required (so the necessary incantations to silence the linter are cheap on average) and since the former is almost always a bug prone to making shutdown/iteration/etc harder than they should be. When I add that rule at new companies, ~90% of the violations are latent bugs. AI has the potential to (though I haven't seen it work well yet in this capacity) lint most such problematic patterns. In an ideal world, it'd even have the local context to know that, e.g., since you're handling a DB transaction and immediately re-throwing the raw `except:` is appropriate and not even flag it in the "review" (the AI linting), reducing the false positive rate. You'd still want a human review, but you could avoid bugging a human till the code is worth reviewing, or you could let the human focus on things that matter instead of harping on your use of low-level atomic fences yet again. AI has potential to improve the transition from "code complete" to "shipped and working."
  - siscia 2y ago
    Wouldn't be simpler to just use a linter then?
    - stavros 2y ago
      It would be simpler, but it wouldn't be as effective.
      - abenga 2y ago
        How so? The build fails if linter checks don't pass. What does an ai add to this?
        stavros 2y ago
        Much more nuanced linting rules.
        2y ago
- eterm 2y ago
  Code review is also one of the last places to get adequate knowledge-sharing into many processes which have gone fully remote / asynchronous.

dakshgupta 2y ago

I haven’t explored Korbit but with CodeRabbit there are a couple of things: 1. We are better at full codebase context, because of how we index the codebase like a graph and use graph search and an LLM to determine what other parts of the codebase should be taken into consideration while reviewing a diff. 2. We offer an API you can use to build custom workflows, for example every time a test fails in your pipeline you can pass the output to Greptile and it will diagnose with full codebase context. 3. Customers of ours that switched from CodeRabbit usually say Greptile has far fewer and less verbose comments. This is a subjective, of course. That said, CodeRabbit is cheaper. Both products have free trials for though, so I would recommend trying both and seeing which one your team prefers, which is ultimately what matters. I’m happy to answer more questions or a demo for your team too -> daksh@greptile.com

swells34 2y ago

Isn't code review the one area you'd never want to replace the programmer in? Like, that is the most important step of the process; it's where we break down the logic and sanity check the implementation. That is expressly where AI is the weakest.

imoverclocked 2y ago
Many code review products are not AI. They are often collections of rules that have been specifically crafted to catch bad patterns. Some of the rules are stylistic in nature and those tend to turn people away the fastest IMHO. Many of these tools can be integrated into a local workflow so they will never ping you on a PR but many developers prefer to just write some code and let the review process suss things out.
yorwba 2y ago
You probably shouldn't take a code review bot's "LGTM!" at face value, but if it tells you there is an issue with your code and it is indeed an issue with your code, it has successfully subbed in for your coworker who didn't get around to looking at it yet.
- imiric 2y ago
  > if it tells you there is an issue with your code and it is indeed an issue with your code That's the crucial bit. If it ends up hallucinating issues as LLMs tend to do[1], then it just adds more work for human developers to confirm its report. Human devs can create false reports as well, but we're more likely to miss an issue than misunderstand and be confident about it. [1]: https://www.theregister.com/2024/12/10/ai_slop_bug_reports/ https://www.theregister.com/2024/12/10/ai_slop_bug_reports/
dakshgupta 2y ago
I agree - at least with where technology is today, I would strongly discourage replacing human code review with AI. It does however serve as a good first pass, so by the time the human reviewer gets it, the little things have been addressed and they can focus on broader technical decisions. Would you want your coworkers to run their PRs through an AI reviewer, resolve those comments, then send it to you?
- deleted 2y ago
  [deleted]
- AgentOrange1234 2y ago
  This feels like the right take. Once we have an AI starting to do something, there’s an art to gradually adopting it in a way that makes sense. We don’t lose the ability to have humans review code. We don’t have to all use this on every PR. We don’t have to blindly accept every comment it makes.
jerrygoyal 2y ago
No one is blindly merging PR after AI Code Review. Think of review bot as an extra pair of eyes.
- bobnamob 2y ago
  Yet. The things I've seen juniors try to do because "the AI said so" is staggering. Furthermore, I have little faith that the average junior gets competent enough teaching/mentoring that this attitude won't be a pervasive problem in 5ish years. Admittedly, maybe I'm getting old and this is just another "darn kids these days" rant
hansvm 2y ago
If you think of the "AI" benefit as more of a linter then the value starts to become clear. E.g., I've never hesitated to add a linting rule requiring `except Exception:` in lieu of `except:` in Python, since the latter is very rarely required (so the necessary incantations to silence the linter are cheap on average) and since the former is almost always a bug prone to making shutdown/iteration/etc harder than they should be. When I add that rule at new companies, ~90% of the violations are latent bugs. AI has the potential to (though I haven't seen it work well yet in this capacity) lint most such problematic patterns. In an ideal world, it'd even have the local context to know that, e.g., since you're handling a DB transaction and immediately re-throwing the raw `except:` is appropriate and not even flag it in the "review" (the AI linting), reducing the false positive rate. You'd still want a human review, but you could avoid bugging a human till the code is worth reviewing, or you could let the human focus on things that matter instead of harping on your use of low-level atomic fences yet again. AI has potential to improve the transition from "code complete" to "shipped and working."
- siscia 2y ago
  Wouldn't be simpler to just use a linter then?
  - stavros 2y ago
    It would be simpler, but it wouldn't be as effective.
    - abenga 2y ago
      How so? The build fails if linter checks don't pass. What does an ai add to this?
      - stavros 2y ago
        Much more nuanced linting rules.
        2y ago
eterm 2y ago
Code review is also one of the last places to get adequate knowledge-sharing into many processes which have gone fully remote / asynchronous.