4 ms·

Creator of Codeball here, somebody beat us to sharing it :). Codeball is a result of a hack-week at Sturdy - we were thinking about ways to reduce the waiting-

Creator of Codeball here, somebody beat us to sharing it :). Codeball is a result of a hack-week at Sturdy - we were thinking about ways to reduce the waiting-for-code-review and were curious exactly how predictable the entire process is. It turned out very predictable! Happy to answer any questions.

cobbal 4y ago
How much will you be charging for the adversarial network to allow someone to get any PR approved? ;)
sidlls 4y ago
I know for a fact I would not want to automate many of the predictable aspects of code reviews at any job I've ever had. This is because many of the predictable aspects of code review are due to poor review practices. Things like rubber-stamping review requests with a certain change size (e.g. lines of code, number of files), surface-level reviews (e.g. "yep this doesn't violate our style guidelines that the linter can't catch"), and similar items. A proper code review isn't simply catching API or style errors--it seeks to understand how the change affects the architecture and structure of the existing code. I'm sure AI can help with that, and for a broad class of changes it's likely somewhat to very predictable--but I'm skeptical that it is predictable for enough use cases to make it worth spending money on (for now), say. Put another way: "approves code reviews a human would have approved" isn't exactly the standard I'd want automated reviews to aspire to. Human approval, in my experience, is mostly not good quality reviews.
- videlov 4y ago
  My thesis is a tool like Codeball would reduce the amount of rubber-stamping. Thing is, many devs aim to ship code "small and often" which inevitably leads to fast pattern matching style reviews. If software can reliably deal with those, humans can focus their energy on the tricky ones. Kind of like how using a code formatter eliminates this type of discussions and lets people focus on semantics.
- visarga 4y ago
  Maybe the AI approach is still useful. I am thinking analysing the AST to measure the impact of a code change, or the complexity of the various components of the project. Some kind of graph analysis to measure complexity and maintainability on a project level.
dchichkov 4y ago
Hi. I've tried creating the same service, about 5 years back ;) articoder.com ;) Was digging at it for a few months. But natural language processing of the time was not up to the task... Good to know that now it is doable in a week, with such good precision! Or do you have humans in the backend ;) ? How do you compare yourself to PullRequest (they've been digging at it for 5 years as well) and recently folded? [funny fact, we've been interviewed in the same YC batch, which always makes me wonder, if YC liked the idea enough to have it implemented by another team ;) ]
- videlov 4y ago
  It's really cool to hear that others have thought about this too! >How do you compare yourself to PullRequest So it turns out that the most of code contributions nowadays get merged without fixes or feedback during the review (about 2/3). I think this is because the increased focus on continuous delivery and shipping small & often. Codeball's purpose is to identify and approve those 'easy' PRs and humans get to deal with the trickier ones. The cool part about it is being less blocked.
emeraldd 4y ago
Is your model trained per language? Without something that semantically understands the code under review ( which all but requires general AI or at the least a strong static analyslzer) doing anything more than adding noise to the process or worse leading to certain groups of developers effectively being given a free pass.
- videlov 4y ago
  It is not trained per language but it has 2 things up its sleeve: it considers the author's past experience in the context of the files being changed as well as if similar code changes (perceptual hashes) are associated with objections or fixes.
  - emeraldd 4y ago
    Both of those statements convince me even more that this is a bad idea. While the author's past experience is important, it has little bearing on the current PR. Same for similar code change. In code review, the skill/history of the developer is only really relevant when writing comments. You should look for the same potential mistakes and logic errors in a Senior or Junior developers PR. Adding the developers experience as an input could easily lead to the model deferring to experience. In my mind, that makes the signal this is providing potentially harmful, not helpful.
RhysU 4y ago
> we were thinking about ways to reduce the waiting-for-code-review Code reviews should be an interrupt for everything except downtime mitigation. Reviewing your peer's code quickly will cause them to do the same to you. It is a virtuous circle. Be the change you want to see.

cobbal 4y ago

How much will you be charging for the adversarial network to allow someone to get any PR approved? ;)

sidlls 4y ago

I know for a fact I would not want to automate many of the predictable aspects of code reviews at any job I've ever had. This is because many of the predictable aspects of code review are due to poor review practices. Things like rubber-stamping review requests with a certain change size (e.g. lines of code, number of files), surface-level reviews (e.g. "yep this doesn't violate our style guidelines that the linter can't catch"), and similar items. A proper code review isn't simply catching API or style errors--it seeks to understand how the change affects the architecture and structure of the existing code. I'm sure AI can help with that, and for a broad class of changes it's likely somewhat to very predictable--but I'm skeptical that it is predictable for enough use cases to make it worth spending money on (for now), say. Put another way: "approves code reviews a human would have approved" isn't exactly the standard I'd want automated reviews to aspire to. Human approval, in my experience, is mostly not good quality reviews.

videlov 4y ago
My thesis is a tool like Codeball would reduce the amount of rubber-stamping. Thing is, many devs aim to ship code "small and often" which inevitably leads to fast pattern matching style reviews. If software can reliably deal with those, humans can focus their energy on the tricky ones. Kind of like how using a code formatter eliminates this type of discussions and lets people focus on semantics.
visarga 4y ago
Maybe the AI approach is still useful. I am thinking analysing the AST to measure the impact of a code change, or the complexity of the various components of the project. Some kind of graph analysis to measure complexity and maintainability on a project level.

dchichkov 4y ago

Hi. I've tried creating the same service, about 5 years back ;) articoder.com ;) Was digging at it for a few months. But natural language processing of the time was not up to the task... Good to know that now it is doable in a week, with such good precision! Or do you have humans in the backend ;) ? How do you compare yourself to PullRequest (they've been digging at it for 5 years as well) and recently folded? [funny fact, we've been interviewed in the same YC batch, which always makes me wonder, if YC liked the idea enough to have it implemented by another team ;) ]

videlov 4y ago
It's really cool to hear that others have thought about this too! >How do you compare yourself to PullRequest So it turns out that the most of code contributions nowadays get merged without fixes or feedback during the review (about 2/3). I think this is because the increased focus on continuous delivery and shipping small & often. Codeball's purpose is to identify and approve those 'easy' PRs and humans get to deal with the trickier ones. The cool part about it is being less blocked.

emeraldd 4y ago

Is your model trained per language? Without something that semantically understands the code under review ( which all but requires general AI or at the least a strong static analyslzer) doing anything more than adding noise to the process or worse leading to certain groups of developers effectively being given a free pass.

videlov 4y ago
It is not trained per language but it has 2 things up its sleeve: it considers the author's past experience in the context of the files being changed as well as if similar code changes (perceptual hashes) are associated with objections or fixes.
- emeraldd 4y ago
  Both of those statements convince me even more that this is a bad idea. While the author's past experience is important, it has little bearing on the current PR. Same for similar code change. In code review, the skill/history of the developer is only really relevant when writing comments. You should look for the same potential mistakes and logic errors in a Senior or Junior developers PR. Adding the developers experience as an input could easily lead to the model deferring to experience. In my mind, that makes the signal this is providing potentially harmful, not helpful.

RhysU 4y ago

> we were thinking about ways to reduce the waiting-for-code-review Code reviews should be an interrupt for everything except downtime mitigation. Reviewing your peer's code quickly will cause them to do the same to you. It is a virtuous circle. Be the change you want to see.