3 ms·

> Judging is considerably easier than writing something from scratch I don't agree with this at all. Writing new code is trivially easy, to do a full in depth

> Judging is considerably easier than writing something from scratch I don't agree with this at all. Writing new code is trivially easy, to do a full in depth review takes significantly more brain power. You have to fully ascertain and insert yourself into someone elses thought process. Thats way more work than utilizing your own thought process.

Herring 1y ago
Sorry, I should have been more specific. I meant LLMs are more reliable and accurate at judging than at generating from scratch. They basically achieve over 80% agreement with human evaluators [1]. This level of agreement is similar to the consensus rate between two human evaluators, making LLM-as-a-judge a scalable and reliable proxy for human judgment. [1] https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2306.05685 (2023)
- habinero 1y ago
  80% is a pretty abysmal success rate and means it's very unreliable. It sounds nice but it means at least 1 in 5 are bad. That's worse odds than rolling 1 on a d6. You'll be tripping over mistakes constantly.
  - andrekandre 1y ago
    > 80% is a pretty abysmal success rate and means it's very unreliable. EXACTLY. imagine if your car would at best start 80% of the time...
    - bluedel 1y ago
      At least if the car doesn't start, you can immediately tell.
      - yencabulator 1y ago
        Imagine Google Maps driving directions took you in the correct direction 80% of the time.
  - PeterStuer 1y ago
    From the study: " [80% is] the same level of agreement between humans." So if 80% is abysmal, you are getting the same abysmal level from human code reviewers.
    - habinero 1y ago
      Not the same thing. With human code reviewers, you can talk to them and resolve conflicts. You can even ask them to review code differently in future. The whole point of a tool is to reduce workload. If you have to frequently correct it, then it's not worthwhile to use.
- malfist 1y ago
  LLMs will not have the context behind the lines of code in the CR. Sure there's no bug with how the logic is defined in the CR or even in the context of the project, ti maybe won't throw an exception. But the LLM won't know that the query is iterating over an unindexed field in the DB with the table in prod having 10s of millions of rows. The LLM won't know that even though the code says the button should be red and the comments say the button should be red, the corporate style guide says red should be a very specific hex code that it isn't.

Herring 1y ago

Sorry, I should have been more specific. I meant LLMs are more reliable and accurate at judging than at generating from scratch. They basically achieve over 80% agreement with human evaluators [1]. This level of agreement is similar to the consensus rate between two human evaluators, making LLM-as-a-judge a scalable and reliable proxy for human judgment. [1] https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2306.05685 (2023)

habinero 1y ago
80% is a pretty abysmal success rate and means it's very unreliable. It sounds nice but it means at least 1 in 5 are bad. That's worse odds than rolling 1 on a d6. You'll be tripping over mistakes constantly.
- andrekandre 1y ago
  > 80% is a pretty abysmal success rate and means it's very unreliable. EXACTLY. imagine if your car would at best start 80% of the time...
  - bluedel 1y ago
    At least if the car doesn't start, you can immediately tell.
    - yencabulator 1y ago
      Imagine Google Maps driving directions took you in the correct direction 80% of the time.
- PeterStuer 1y ago
  From the study: " [80% is] the same level of agreement between humans." So if 80% is abysmal, you are getting the same abysmal level from human code reviewers.
  - habinero 1y ago
    Not the same thing. With human code reviewers, you can talk to them and resolve conflicts. You can even ask them to review code differently in future. The whole point of a tool is to reduce workload. If you have to frequently correct it, then it's not worthwhile to use.
malfist 1y ago
LLMs will not have the context behind the lines of code in the CR. Sure there's no bug with how the logic is defined in the CR or even in the context of the project, ti maybe won't throw an exception. But the LLM won't know that the query is iterating over an unindexed field in the DB with the table in prod having 10s of millions of rows. The LLM won't know that even though the code says the button should be red and the comments say the button should be red, the corporate style guide says red should be a very specific hex code that it isn't.