3 ms·

Agree with that. Read it as expert-level knowledge without all the other stuff LLMs can’t do as well as humans. LLMs way to express knowledge is kinda of alien

Agree with that. Read it as expert-level knowledge without all the other stuff LLMs can’t do as well as humans. LLMs way to express knowledge is kinda of alien as it is different, so indeed those are all poor simplifications. For instance an LLM can’t code as well as a top human coder but can write a non trivial program from the first to the last character without iterating.

spyckie2 1y ago
Hey antirez, What sticks out to me is Gemini catching bugs before production release, was hoping you’d give a little more insight into that. Reason being is that we expect ai to create bugs and we catch them, but if Gemini is spotting bugs by some way of it being a QA (not just by writing and passing tests) then that perks my interest.
- jacobr1 1y ago
  Our team has pretty aggressively started using LLMs for automated code review. It will look at our PRs and post comments. We can adding more material for different things for it to consider- from a looking at a summarized version of our API guidelines, general prompts like, "You are an expert software engineer and QA professional, review this PR and point out any bugs or other areas of technical risk. Make concise suggestions for improvement where applicable." - it catches a ton of stuff. Another area we've started doing is having it look at build failures and writing a report on suggested root causes before even a human looks at it - saves time. Or (and we haven't rolled this out automatically yet but are testing a prototype) having it triage alarms from our metrics, with access to the logs and codebase to investigate.
  - infecto 1y ago
    I have been surprised more folks have no rolled these out as paid for products. I have been getting tons of use out of systems like cursors bugbot. The signal to noise is high and while it’s not always right it catches a lot of bugs I would have missed.
    - senko 1y ago
      There are a few: Greptile, Ellipsis, GH Copilot (integrated with GH) I feel many also try "review and fix automatically", as it's tempting to "just" pass the generated comments to a second agent to apply them. But that opens a whole other can of worms and pretty soon you're just another code assistant service.
      - dearilos 1y ago
        if you do specific prompts based on your team's tribal knowledge and standards it works really well "look at this code for bugs" doesn't end up working well, which is what most code reviewers do.
        jacobr1 1y ago
        Yep - this is the key to making it work
  - dearilos 1y ago
    how are you dealing with the accuracy of the review comments? in my experience "review this PR" is very generic and ends up giving slop.
    - jacobr1 1y ago
      Yeah - that doesn't work. We include our own guidelines and practices in the context. Basically an ever evolving wiki page of PR best practices. We had that before trying LLMs - so it was easier for us to start. Also we found doing an LLM reformat of that data to might a much tighter set of "rules" helped as well.
      - dearilos 1y ago
        ive been building out a directory of code review rules for the last couple of months! are you open to chatting and sharing notes on what works/doesn't work? my email is ilya (at) wispbit.com

spyckie2 1y ago

Hey antirez, What sticks out to me is Gemini catching bugs before production release, was hoping you’d give a little more insight into that. Reason being is that we expect ai to create bugs and we catch them, but if Gemini is spotting bugs by some way of it being a QA (not just by writing and passing tests) then that perks my interest.

jacobr1 1y ago
Our team has pretty aggressively started using LLMs for automated code review. It will look at our PRs and post comments. We can adding more material for different things for it to consider- from a looking at a summarized version of our API guidelines, general prompts like, "You are an expert software engineer and QA professional, review this PR and point out any bugs or other areas of technical risk. Make concise suggestions for improvement where applicable." - it catches a ton of stuff. Another area we've started doing is having it look at build failures and writing a report on suggested root causes before even a human looks at it - saves time. Or (and we haven't rolled this out automatically yet but are testing a prototype) having it triage alarms from our metrics, with access to the logs and codebase to investigate.
- infecto 1y ago
  I have been surprised more folks have no rolled these out as paid for products. I have been getting tons of use out of systems like cursors bugbot. The signal to noise is high and while it’s not always right it catches a lot of bugs I would have missed.
  - senko 1y ago
    There are a few: Greptile, Ellipsis, GH Copilot (integrated with GH) I feel many also try "review and fix automatically", as it's tempting to "just" pass the generated comments to a second agent to apply them. But that opens a whole other can of worms and pretty soon you're just another code assistant service.
    - dearilos 1y ago
      if you do specific prompts based on your team's tribal knowledge and standards it works really well "look at this code for bugs" doesn't end up working well, which is what most code reviewers do.
      - jacobr1 1y ago
        Yep - this is the key to making it work
- dearilos 1y ago
  how are you dealing with the accuracy of the review comments? in my experience "review this PR" is very generic and ends up giving slop.
  - jacobr1 1y ago
    Yeah - that doesn't work. We include our own guidelines and practices in the context. Basically an ever evolving wiki page of PR best practices. We had that before trying LLMs - so it was easier for us to start. Also we found doing an LLM reformat of that data to might a much tighter set of "rules" helped as well.
    - dearilos 1y ago
      ive been building out a directory of code review rules for the last couple of months! are you open to chatting and sharing notes on what works/doesn't work? my email is ilya (at) wispbit.com