4 ms·

>there is no point in code-reviewing ai-generated code the idea that you should just blindly trust code you are responsible for without bothering to review it

by knome 6mo ago

>there is no point in code-reviewing ai-generated code the idea that you should just blindly trust code you are responsible for without bothering to review it is ludicrous.

eclipxe 6mo ago
Why?
- fl4regun 6mo ago
  Is this a serious question? If you are handling sensitive information how do you confirm your application is secure and won't leak or expose information to people who shouldn't know it?
  - lijok 6mo ago
    How do you with classic code?
    - hallway_monitor 6mo ago
      Exactly.... -> Unit tests. Integration tests. UI tests. This is how code should be verified no matter the author. Just today I told my team we should not be reading every line of LLM code. Understand the pattern. Read the interesting / complex parts. Read the tests.
      - GrinningFool 6mo ago
        But unit and integration tests generally only catch the things you can think of. That leaves a lot of unexplored space in which things can go wrong. Separately, but related - if you offload writing of the tests and writing of the code, how does anybody know what they have other than green tests and coverage numbers?
        dntrkv 6mo ago
        I have been seeing this problem building over the last year. LLM generated logic being tested by massive LLM generated tests. Everyone just goes overboard with the tests since you can easily just tell the LLM to expand on the suite. So you end up with a massive test suite that looks very thorough and is less likely to be scrutinized.
    - fl4regun 6mo ago
      if you are asking me how you *guarantee* there is not a single possible exploit in your code, you can't do that. But you can do your best and learn about common pitfalls and be reasonably competent. Just because you can't do the former doesn't mean the latter is useless.
jen20 6mo ago
(I mostly agree with you, but) devils advocate: most people already do that with dependencies, so why not move the line even further up?
- batshit_beaver 6mo ago
  Because you trust that your dependencies are not vibe coded and have been reviewed by humans.
  - bdangubic 6mo ago
    except they are vibe-or-not coded by some dude in Reno NV who wouldn’t pass a phone screen where you work
    - batshit_beaver 6mo ago
      I'd trust that dude over professional leetcoders any day. But you're right that trust is a complicated thing and often misplaced. I think as an industry we're always reevaluating our relationship with OSS, and I'm sure LLMs will affect this relationship in some way. It's too early to tell.
      - bdangubic 6mo ago
        I find this relationship fascinating. since the OSS vast majority of the developers will not hesitate to pull in library X or framework Y knowing really nothing about it, who are developers, what is the quality of it, what is their release process, qa etc etc... The first thing I do now as a "senior" for decades when I get approached with "we should consider using ____" is to send them to their issues page ( e.g. https://github.com/oven-sh/bun/issues https://github.com/oven-sh/bun/issues ) and then be like "spend 60-90 minutes minimum here reviewing the issues - then come back and tell me whether or not the inclusion of this is something we should consider." and yet, now with LLMs there are sooooooooo many comments on HN like "oh they must be supervised, who knows what they will be doing etc..." - gotta supervise them but some mate in Boise is all good, hopefully someone else will review his stuff that is going into your next release ...
ramraj07 6mo ago
You are still responsible for the product; the code has stopped being what defines the product.
- Uhhrrr 6mo ago
  If you don't review what the product does, you are irresponsible for the product.
  - jmalicki 6mo ago
    Is the CEO responsible for a company's financial performance? Do they review every line of code the company writes? It is more irresponsible to spend the time reviewing all of the code rather than spending that time on things with bigger levers for satisfying your customers.
    - NothingAboutAny 6mo ago
      yes but if a dev pushes a line of code that wipes the accounts of millions of users at a fintech, the dev will get fired but the CEO will get sued into oblivion. if the agent isn't responsible, you HAVE to be, cause angry people wont listen to "it's no ones fault your money is gone"

by knome 6mo ago

>there is no point in code-reviewing ai-generated code the idea that you should just blindly trust code you are responsible for without bothering to review it is ludicrous.

eclipxe 6mo ago

Why?

fl4regun 6mo ago
Is this a serious question? If you are handling sensitive information how do you confirm your application is secure and won't leak or expose information to people who shouldn't know it?
- lijok 6mo ago
  How do you with classic code?
  - hallway_monitor 6mo ago
    Exactly.... -> Unit tests. Integration tests. UI tests. This is how code should be verified no matter the author. Just today I told my team we should not be reading every line of LLM code. Understand the pattern. Read the interesting / complex parts. Read the tests.
    - GrinningFool 6mo ago
      But unit and integration tests generally only catch the things you can think of. That leaves a lot of unexplored space in which things can go wrong. Separately, but related - if you offload writing of the tests and writing of the code, how does anybody know what they have other than green tests and coverage numbers?
      - dntrkv 6mo ago
        I have been seeing this problem building over the last year. LLM generated logic being tested by massive LLM generated tests. Everyone just goes overboard with the tests since you can easily just tell the LLM to expand on the suite. So you end up with a massive test suite that looks very thorough and is less likely to be scrutinized.
  - fl4regun 6mo ago
    if you are asking me how you *guarantee* there is not a single possible exploit in your code, you can't do that. But you can do your best and learn about common pitfalls and be reasonably competent. Just because you can't do the former doesn't mean the latter is useless.

jen20 6mo ago

(I mostly agree with you, but) devils advocate: most people already do that with dependencies, so why not move the line even further up?

batshit_beaver 6mo ago
Because you trust that your dependencies are not vibe coded and have been reviewed by humans.
- bdangubic 6mo ago
  except they are vibe-or-not coded by some dude in Reno NV who wouldn’t pass a phone screen where you work
  - batshit_beaver 6mo ago
    I'd trust that dude over professional leetcoders any day. But you're right that trust is a complicated thing and often misplaced. I think as an industry we're always reevaluating our relationship with OSS, and I'm sure LLMs will affect this relationship in some way. It's too early to tell.
    - bdangubic 6mo ago
      I find this relationship fascinating. since the OSS vast majority of the developers will not hesitate to pull in library X or framework Y knowing really nothing about it, who are developers, what is the quality of it, what is their release process, qa etc etc... The first thing I do now as a "senior" for decades when I get approached with "we should consider using ____" is to send them to their issues page ( e.g. https://github.com/oven-sh/bun/issues https://github.com/oven-sh/bun/issues ) and then be like "spend 60-90 minutes minimum here reviewing the issues - then come back and tell me whether or not the inclusion of this is something we should consider." and yet, now with LLMs there are sooooooooo many comments on HN like "oh they must be supervised, who knows what they will be doing etc..." - gotta supervise them but some mate in Boise is all good, hopefully someone else will review his stuff that is going into your next release ...

ramraj07 6mo ago

You are still responsible for the product; the code has stopped being what defines the product.

Uhhrrr 6mo ago
If you don't review what the product does, you are irresponsible for the product.
- jmalicki 6mo ago
  Is the CEO responsible for a company's financial performance? Do they review every line of code the company writes? It is more irresponsible to spend the time reviewing all of the code rather than spending that time on things with bigger levers for satisfying your customers.
  - NothingAboutAny 6mo ago
    yes but if a dev pushes a line of code that wipes the accounts of millions of users at a fintech, the dev will get fired but the CEO will get sued into oblivion. if the agent isn't responsible, you HAVE to be, cause angry people wont listen to "it's no ones fault your money is gone"