6 ms·

I don't understand the premise. If I trust someone to write good code, I learned to trust them because their code works well, not because I have a theory of min

I don't understand the premise. If I trust someone to write good code, I learned to trust them because their code works well, not because I have a theory of mind for them that "produces good code" a priori. If someone uses an LLM and produces bug-free code, I'll trust them. If someone uses an LLM and produces buggy code, I won't trust them. How is this different from when they were only using their brain to produce the code?

moffkalast 1y ago
It's easy to get overconfident and not test the LLM's code enough when it worked fine for a handful of times in a row, and then you miss something. The problem is often really one of miscommunication, the task may be clear to the person working on it, but with frequent context resets it's hard to make sure the LLM also knows what the whole picture is and they tend to make dumb assumptions when there's ambiguity. The thing that 4o does with deep research where it asks for additional info before it does anything should be standard for any code generation too tbh, it would prevent a mountain of issues.
- stavros 1y ago
  Sure, but you're still responsible for the quality of the code you commit, LLM or no.
  - moffkalast 1y ago
    Of course you are, but it's sort of like how people are responsible their Tesla driving on autopilot, which then suddenly swerves into a wall and disengages two seconds before impact. The process forces you to make mistakes you wouldn't normally ever do or even consider a possibility.
    - JohnKemeny 1y ago
      To add to devs and Teslas, you have journalists using LLMs writing summaries, lawyers using LLMs writing dispositions, doctors using LLMs writing their patient entries, and law enforcement using LLMs writing their forensics report. All of these make mistakes (there are documented incidents). And yes, we can counter with "the journalists are dumb for not verifying", "the lawyers are dumb for not checking", etc., but we should also be open for the fact that these are intelligent and professional people who make mistakes because they were mislead by those who sell LLMs.
      - bluefirebrand 1y ago
        I think it's analogous to physical labour In the past someone might have been physically healthy and strong enough to physically shovel dirt all day long Nowadays this is rarer because we use an excavator instead. Yes, a professional dirt mover is more productive with an excavator than a shovel, but is likely not as physically fit as someone spending their days moving dirt with a shovel I think it will be similar with AI. It is absolutely going to offload a lot of people's thinking into the LLMs and their "do it by hand" muscles will atrophy. For knowledge workers, that's our brain I know this was a similar concern with search engines and Stack Overflow, so I am trying to temper my concern here as best I can. But I can't shake the feeling that LLMs provide a way for people to offload their thinking and go on autopilot a lot more easily than Search ever did I'm not saying that we were better off when we had to move dirt by hand either. I'm just saying there was a physical tradeoff when people moved out of the fields and into offices. I suspect there will be a cognitive tradeoff now that we are moving away from researching solutions to problems and towards asking the AI to give us solutions to problems
  - acedTrex 1y ago
    In an ideal world you would think everyone see's it this way. But we are starting to see an uptick in "I don't know the LLMs said do that." As if that is a somehow exonerating sentence.
    - NeutralCrane 1y ago
      It isn’t, and that is a sign of a bad dev you shouldn’t trust. LLMs are a tool, just like any number of tools that are used by developers in modern software development. If a dev doesn’t use the tool properly, don’t trust them. If they do, trust them. The way to assess if they use it properly is in the code they produce. Your premise is just fundamentally flawed. Before LLMs, the proof of a quality dev was in the pudding. After LLMs, the proof of a quality dev remains in the pudding.
      - acedTrex 1y ago
        > Your premise is just fundamentally flawed. Before LLMs, the proof of a quality dev was in the pudding. After LLMs, the proof of a quality dev remains in the pudding. Indeed it does, however what the "proof" is has changed. In terms of sitting down and doing a full, deep review, tracing every path validating every line etc. Then for sure, nothing has changed. However, at least in my experience, pre LLM those reviews were not EVERY CASE there were many times I elided parts of a deep review because i saw markers in the code that to me showed competency, care etc. With those markers there are certain failure conditions that can be deemed very unlikely to exist and therefore the checks can be skipped. Is that ALWAYS the correct assumption? Absolutely not but the more experienced you are the less false positives you get. LLMs make those markers MUCH harder to spot, so you have to fall back to doing a FULL indepth review no matter what. You have to eat ALL the pudding so to speak. For people that relied on maybe tasting a bit of the pudding then assuming based on the taste the rest of the pudding probably tastes the same its rather jarring and exhausting to now have to eat all of it all the time.
        NeutralCrane 1y ago
        > However, at least in my experience, pre LLM those reviews were not EVERY CASE there were many times I elided parts of a deep review because i saw markers in the code that to me showed competency, care etc. That was never proof in the first place. If anything, someone basing their trust in a submission on anything other than the code itself is far more concerning and trust-damaging to me than if the submitter has used an LLM.
taneq 1y ago
If you have a long standing, effective heuristic that “people with excellent, professional writing are more accurate and reliable than people with sloppy spelling and punctuation” then the appearance of a semi-infinite group of ‘people’ writing well presented, convincingly worded articles which nonetheless are riddled with misinformation, hidden logical flaws, and inconsistencies, you’re gonna end up trusting everyone a lot less. It’s like if someone started bricking up tunnel entrances and painting ultra realistic versions of the classic Road Runner tunnel painting on them, all over the place. You’d have to stop and poke every underpass with a stick just to be sure.
- stavros 1y ago
  Sure, your heuristic no longer works, and that's a bit inconvenient. We'll just find new ones.
  - sebmellen 1y ago
    Yeah, now you need to be able to demonstrate verbal fluency. The problem is, that inherently means a loss of “trusted anonymous” communication, which is particularly damaging to the fiber of the internet.
    - acedTrex 1y ago
      Author here: Precisely, in the age where it is very difficult to ascertain the type or quality of skills you are interacting with say in a patch review or otherwise you frankly have to "judge" someone and fallback to suspicion and full verification.
    - taneq 1y ago
      Yeah I think "trust for a fluent, seemingly logically coherent anonymous responder" pretty much captures it.
  - oasisaimlessly 1y ago
    "A bit inconvenient" might be the understatement of the year. If information requires say, 2x the time to validate, the utility of the internet is halved.
alganet 1y ago
> I learned to trust them because their code works well There's so much more than "works well". There are many cues that exist close to code, but are not code: I trust more if the contributor explains their change well. I trust more if the contributor did great things in the past. I trust more if the contributor manages granularity well (reasonable commits, not huge changes). I trust more if the contributor picks the right problems to work on (fixing bugs before adding new features, etc). I trust more if the contributor proves being able to maintain existing code, not just add on top of it. I trust more if the contributor makes regular contributions. And so on...
- acedTrex 1y ago
  Author here: Spot on, there are so many little things that we as humans use as subtle verification steps to decide how much scrutiny various things require. LLMs are not necessarily the death of that concept but they do make it far far harder.
somewhereoutth 1y ago
Because when people use LLMs, they are getting the tool to do the work for them, not using the tool to do the work. LLMs are not calculators, nor are they the internet. A good rule of thumb is to simply reject any work that has had involvement of an LLM, and ignore any communication written by an LLM (even for EFL speakers, I'd much rather have your "bad" English than whatever ChatGPT says for you). I suspect that as the serious problems with LLMs become ever more apparent, this will become standard policy across the board. Certainly I hope so.
- stavros 1y ago
  Well, no, a good rule of thumb is to expect people to write good code, no matter how they do it. Why would you mandate what tool they can use to do it?
  - somewhereoutth 1y ago
    Because it pertains to the quality of the output - I can't validate every line of code, or test every edge case. So if I need a certain level of quality, I have to verify the process of producing it. This is standard for any activity where accuracy / safety is paramount - you validate the process. Hence things like maintenance logs for airplanes.
    - mexicocitinluez 1y ago
      [flagged]
    - acedTrex 1y ago
      > So if I need a certain level of quality, I have to verify the process of producing it Precisely this, and this is hardly a unique to software requirement. Process audits are everywhere in engineering. Previously you could infer the process of producing some code by simply reading the patch and that generally would tell you quite a bit about the author itself. Using advanced and niche concepts with imply a solid process with experience backing it. Which would then imply that certain contextual bugs are unlikely so you skip looking for them. My premise in the blog is basically that "Well now I have go do a full review no matter what the code itself tells me about the author."
mexicocitinluez 1y ago
It's not. What you're seeing now is people who once thought and proclaimed these tools as useless now have to start to walk back their claims with stuff like this. It does amaze me that the people who don't use these tools seem to have the most to say about them.
- acedTrex 1y ago
  Author here: For what it's worth I do actually use the tools albeit incredibly intentionally and sparingly. I see quite a few workflows and tasks that they can be a value add on, mostly outside of the hotpath of actual code generation but still quite enticing. So much so in fact I'm working on my own local agentic tool with some self hosted ollama models. I like to think that i am at least somewhat in the know on the capabilities and failure points of the latest LLM tooling. That however doesn't change my thoughts on trying to ascertain if code submitted to me deserves a full indepth review or if I can maybe cut a few corners here and there.
  - mexicocitinluez 1y ago
    > That however doesn't change my thoughts on trying to ascertain if code submitted to me deserves a full indepth review or if I can maybe cut a few corners here and there. How would you even know? Seriously, if I use Chatgpt to generate a one-off function for a feature I'm working on that searches all classes for one that inherits a specific interface and attribute, are you saying you'd be able to spot the difference? And what does it even matter it works? What if I use Bolt to generate a quick screen for a PoC? Or use Claude to create a print-preview with CSS of a 30 page Medicare form? Or converting a component's styles MUI to tailwind? What if all these things are correct? This whole OS repos will ban LLM-generated code is a bit absurd. > or what it's worth I do actually use the tools albeit incredibly intentionally and sparingly. How sparingly? Enough to see how it's constantly improving?
    - acedTrex 1y ago
      > How would you even know? Seriously, if I use Chatgpt to generate a one-off function for a feature I'm working on that searches all classes for one that inherits a specific interface and attribute, are you saying you'd be able to spot the difference? I don't know, thats the problem. As a result, because I can't know I have to now do full in depth reviews no matter what. Which is the "judging" I tongue in cheek talk about in the blog. > How sparingly? Enough to see how it's constantly improving? Nearly daily, to be honest I have not noticed too much improvement year over year in regards to how they fail. They still break in the exact same dumb ways now as they did before. Sure they might generate correct syntactic code reliably now and it might even work. But they still consistently fail to grok the underlying reasoning for things existing. But I am writing my own versions of these agentic systems to use for some rote menial stuff.
acedTrex 1y ago
Author here: Essentially the premise is that in medium trust environments like very large teams or low trust environments like an open source project. LLMs make it very difficult to make an immediate snap judgement about the quality of the dev that submitted the patch based solely on the code itself. In the absence of being able to ascertain the type of person you are dealing with you have to fall back too "no trust" and review everything with a very fine tooth comb. Essentially there are no longer any safe "review shortcuts" and that can be painful in places that relied on those markers to grease the wheels so to speak. Obviously if you are in an existing competent high trust team then this problem does not apply and most likely seems completely foreign as a concept.
- lxgr 1y ago
  > LLMs make it very difficult to make an immediate snap judgement about the quality [...] That's the core of the issue. It's time to say goodbye to heuristics like "the blog post is written in eloquent, grammatical English, hence the point its author is trying to make must be true" or "the code is idiomatic and following all code styles, hence it must be modeling the world with high fidelity". Maybe that's not the worst thing in the world. I feel like it often made people complacent.
  - acedTrex 1y ago
    > Maybe that's not the worst thing in the world. I feel like it often made people complacent. For sure, in some ways perhaps reverting to a low trust environment might improve quality in that it now forces harsher/more in depth reviews. That however doesn't make the requirement less exhausting for people previously relying heavily on those markers to speed things up. Will be very interesting to see how the industry standardizes around this. Right now it's a bit of the wild west. Maybe people in ten years will look back at this post and think "what do you mean you judged people based on the code itself that's ridiculous"
  - furyofantares 1y ago
    I think you're unfair to the heuristics people use in your framing here. You said "hence the point its author is trying to make must be true" and "hence it must be modeling the world with high fidelity". But it's more like "hence the author is likely competent and likely put in a reasonable effort." When those assumptions hold, putting in a very deep review is less likely to pay off. Maybe you are right that people have been too complacent to begin with, I don't know, but I don't think you've framed it fairly.
insane_dreamer 1y ago
> If someone uses an LLM and produces bug-free code, I'll trust them. Only because you already trust them to know that the code is indeed bug-free. Some cases are simple and straightforward -- this routine returns a desired value or it doesn't. Other situations are much more complex in anticipating the ways in which it might interact with other parts of the system, edge cases that are not obvious, etc. Writing code that is "bug free" in that situation requires the writer of the code to understand the implications of the code, and if the dev doesn't understand exactly what the code does because it was written by an LLM, then they won't be able to understand the implications of the code. It then falls to the reviewer to understand the implications of the code -- increasing their workload. That was the premise.
legacynl 1y ago
> How is this different from when they were only using their brain to produce the code? If A = B, higher B means that A is higher too. Now it's A + Ai = B. Now Higher B doesn't necessarily mean higher A. Especially since the current state of Ai is pretty much stochastic, and sometimes is worse than nothing at all
JambalayaJimbo 1y ago
I have never been in a work environment in which I’ve been able to do more than rubber stamp PRs. Performing a deep review of each change is simply impossible with the expectations we were given.
- stavros 1y ago
  Interesting, I've never been in one where we didn't read PRs.
cipherhood 1y ago
[dead]

moffkalast 1y ago

It's easy to get overconfident and not test the LLM's code enough when it worked fine for a handful of times in a row, and then you miss something. The problem is often really one of miscommunication, the task may be clear to the person working on it, but with frequent context resets it's hard to make sure the LLM also knows what the whole picture is and they tend to make dumb assumptions when there's ambiguity. The thing that 4o does with deep research where it asks for additional info before it does anything should be standard for any code generation too tbh, it would prevent a mountain of issues.

stavros 1y ago
Sure, but you're still responsible for the quality of the code you commit, LLM or no.
- moffkalast 1y ago
  Of course you are, but it's sort of like how people are responsible their Tesla driving on autopilot, which then suddenly swerves into a wall and disengages two seconds before impact. The process forces you to make mistakes you wouldn't normally ever do or even consider a possibility.
  - JohnKemeny 1y ago
    To add to devs and Teslas, you have journalists using LLMs writing summaries, lawyers using LLMs writing dispositions, doctors using LLMs writing their patient entries, and law enforcement using LLMs writing their forensics report. All of these make mistakes (there are documented incidents). And yes, we can counter with "the journalists are dumb for not verifying", "the lawyers are dumb for not checking", etc., but we should also be open for the fact that these are intelligent and professional people who make mistakes because they were mislead by those who sell LLMs.
    - bluefirebrand 1y ago
      I think it's analogous to physical labour In the past someone might have been physically healthy and strong enough to physically shovel dirt all day long Nowadays this is rarer because we use an excavator instead. Yes, a professional dirt mover is more productive with an excavator than a shovel, but is likely not as physically fit as someone spending their days moving dirt with a shovel I think it will be similar with AI. It is absolutely going to offload a lot of people's thinking into the LLMs and their "do it by hand" muscles will atrophy. For knowledge workers, that's our brain I know this was a similar concern with search engines and Stack Overflow, so I am trying to temper my concern here as best I can. But I can't shake the feeling that LLMs provide a way for people to offload their thinking and go on autopilot a lot more easily than Search ever did I'm not saying that we were better off when we had to move dirt by hand either. I'm just saying there was a physical tradeoff when people moved out of the fields and into offices. I suspect there will be a cognitive tradeoff now that we are moving away from researching solutions to problems and towards asking the AI to give us solutions to problems
- acedTrex 1y ago
  In an ideal world you would think everyone see's it this way. But we are starting to see an uptick in "I don't know the LLMs said do that." As if that is a somehow exonerating sentence.
  - NeutralCrane 1y ago
    It isn’t, and that is a sign of a bad dev you shouldn’t trust. LLMs are a tool, just like any number of tools that are used by developers in modern software development. If a dev doesn’t use the tool properly, don’t trust them. If they do, trust them. The way to assess if they use it properly is in the code they produce. Your premise is just fundamentally flawed. Before LLMs, the proof of a quality dev was in the pudding. After LLMs, the proof of a quality dev remains in the pudding.
    - acedTrex 1y ago
      > Your premise is just fundamentally flawed. Before LLMs, the proof of a quality dev was in the pudding. After LLMs, the proof of a quality dev remains in the pudding. Indeed it does, however what the "proof" is has changed. In terms of sitting down and doing a full, deep review, tracing every path validating every line etc. Then for sure, nothing has changed. However, at least in my experience, pre LLM those reviews were not EVERY CASE there were many times I elided parts of a deep review because i saw markers in the code that to me showed competency, care etc. With those markers there are certain failure conditions that can be deemed very unlikely to exist and therefore the checks can be skipped. Is that ALWAYS the correct assumption? Absolutely not but the more experienced you are the less false positives you get. LLMs make those markers MUCH harder to spot, so you have to fall back to doing a FULL indepth review no matter what. You have to eat ALL the pudding so to speak. For people that relied on maybe tasting a bit of the pudding then assuming based on the taste the rest of the pudding probably tastes the same its rather jarring and exhausting to now have to eat all of it all the time.
      - NeutralCrane 1y ago
        > However, at least in my experience, pre LLM those reviews were not EVERY CASE there were many times I elided parts of a deep review because i saw markers in the code that to me showed competency, care etc. That was never proof in the first place. If anything, someone basing their trust in a submission on anything other than the code itself is far more concerning and trust-damaging to me than if the submitter has used an LLM.

taneq 1y ago

If you have a long standing, effective heuristic that “people with excellent, professional writing are more accurate and reliable than people with sloppy spelling and punctuation” then the appearance of a semi-infinite group of ‘people’ writing well presented, convincingly worded articles which nonetheless are riddled with misinformation, hidden logical flaws, and inconsistencies, you’re gonna end up trusting everyone a lot less. It’s like if someone started bricking up tunnel entrances and painting ultra realistic versions of the classic Road Runner tunnel painting on them, all over the place. You’d have to stop and poke every underpass with a stick just to be sure.

stavros 1y ago
Sure, your heuristic no longer works, and that's a bit inconvenient. We'll just find new ones.
- sebmellen 1y ago
  Yeah, now you need to be able to demonstrate verbal fluency. The problem is, that inherently means a loss of “trusted anonymous” communication, which is particularly damaging to the fiber of the internet.
  - acedTrex 1y ago
    Author here: Precisely, in the age where it is very difficult to ascertain the type or quality of skills you are interacting with say in a patch review or otherwise you frankly have to "judge" someone and fallback to suspicion and full verification.
  - taneq 1y ago
    Yeah I think "trust for a fluent, seemingly logically coherent anonymous responder" pretty much captures it.
- oasisaimlessly 1y ago
  "A bit inconvenient" might be the understatement of the year. If information requires say, 2x the time to validate, the utility of the internet is halved.

alganet 1y ago

> I learned to trust them because their code works well There's so much more than "works well". There are many cues that exist close to code, but are not code: I trust more if the contributor explains their change well. I trust more if the contributor did great things in the past. I trust more if the contributor manages granularity well (reasonable commits, not huge changes). I trust more if the contributor picks the right problems to work on (fixing bugs before adding new features, etc). I trust more if the contributor proves being able to maintain existing code, not just add on top of it. I trust more if the contributor makes regular contributions. And so on...

acedTrex 1y ago
Author here: Spot on, there are so many little things that we as humans use as subtle verification steps to decide how much scrutiny various things require. LLMs are not necessarily the death of that concept but they do make it far far harder.

somewhereoutth 1y ago

Because when people use LLMs, they are getting the tool to do the work for them, not using the tool to do the work. LLMs are not calculators, nor are they the internet. A good rule of thumb is to simply reject any work that has had involvement of an LLM, and ignore any communication written by an LLM (even for EFL speakers, I'd much rather have your "bad" English than whatever ChatGPT says for you). I suspect that as the serious problems with LLMs become ever more apparent, this will become standard policy across the board. Certainly I hope so.

stavros 1y ago
Well, no, a good rule of thumb is to expect people to write good code, no matter how they do it. Why would you mandate what tool they can use to do it?
- somewhereoutth 1y ago
  Because it pertains to the quality of the output - I can't validate every line of code, or test every edge case. So if I need a certain level of quality, I have to verify the process of producing it. This is standard for any activity where accuracy / safety is paramount - you validate the process. Hence things like maintenance logs for airplanes.
  - mexicocitinluez 1y ago
    [flagged]
  - acedTrex 1y ago
    > So if I need a certain level of quality, I have to verify the process of producing it Precisely this, and this is hardly a unique to software requirement. Process audits are everywhere in engineering. Previously you could infer the process of producing some code by simply reading the patch and that generally would tell you quite a bit about the author itself. Using advanced and niche concepts with imply a solid process with experience backing it. Which would then imply that certain contextual bugs are unlikely so you skip looking for them. My premise in the blog is basically that "Well now I have go do a full review no matter what the code itself tells me about the author."

mexicocitinluez 1y ago

It's not. What you're seeing now is people who once thought and proclaimed these tools as useless now have to start to walk back their claims with stuff like this. It does amaze me that the people who don't use these tools seem to have the most to say about them.

acedTrex 1y ago
Author here: For what it's worth I do actually use the tools albeit incredibly intentionally and sparingly. I see quite a few workflows and tasks that they can be a value add on, mostly outside of the hotpath of actual code generation but still quite enticing. So much so in fact I'm working on my own local agentic tool with some self hosted ollama models. I like to think that i am at least somewhat in the know on the capabilities and failure points of the latest LLM tooling. That however doesn't change my thoughts on trying to ascertain if code submitted to me deserves a full indepth review or if I can maybe cut a few corners here and there.
- mexicocitinluez 1y ago
  > That however doesn't change my thoughts on trying to ascertain if code submitted to me deserves a full indepth review or if I can maybe cut a few corners here and there. How would you even know? Seriously, if I use Chatgpt to generate a one-off function for a feature I'm working on that searches all classes for one that inherits a specific interface and attribute, are you saying you'd be able to spot the difference? And what does it even matter it works? What if I use Bolt to generate a quick screen for a PoC? Or use Claude to create a print-preview with CSS of a 30 page Medicare form? Or converting a component's styles MUI to tailwind? What if all these things are correct? This whole OS repos will ban LLM-generated code is a bit absurd. > or what it's worth I do actually use the tools albeit incredibly intentionally and sparingly. How sparingly? Enough to see how it's constantly improving?
  - acedTrex 1y ago
    > How would you even know? Seriously, if I use Chatgpt to generate a one-off function for a feature I'm working on that searches all classes for one that inherits a specific interface and attribute, are you saying you'd be able to spot the difference? I don't know, thats the problem. As a result, because I can't know I have to now do full in depth reviews no matter what. Which is the "judging" I tongue in cheek talk about in the blog. > How sparingly? Enough to see how it's constantly improving? Nearly daily, to be honest I have not noticed too much improvement year over year in regards to how they fail. They still break in the exact same dumb ways now as they did before. Sure they might generate correct syntactic code reliably now and it might even work. But they still consistently fail to grok the underlying reasoning for things existing. But I am writing my own versions of these agentic systems to use for some rote menial stuff.

acedTrex 1y ago

Author here: Essentially the premise is that in medium trust environments like very large teams or low trust environments like an open source project. LLMs make it very difficult to make an immediate snap judgement about the quality of the dev that submitted the patch based solely on the code itself. In the absence of being able to ascertain the type of person you are dealing with you have to fall back too "no trust" and review everything with a very fine tooth comb. Essentially there are no longer any safe "review shortcuts" and that can be painful in places that relied on those markers to grease the wheels so to speak. Obviously if you are in an existing competent high trust team then this problem does not apply and most likely seems completely foreign as a concept.

lxgr 1y ago
> LLMs make it very difficult to make an immediate snap judgement about the quality [...] That's the core of the issue. It's time to say goodbye to heuristics like "the blog post is written in eloquent, grammatical English, hence the point its author is trying to make must be true" or "the code is idiomatic and following all code styles, hence it must be modeling the world with high fidelity". Maybe that's not the worst thing in the world. I feel like it often made people complacent.
- acedTrex 1y ago
  > Maybe that's not the worst thing in the world. I feel like it often made people complacent. For sure, in some ways perhaps reverting to a low trust environment might improve quality in that it now forces harsher/more in depth reviews. That however doesn't make the requirement less exhausting for people previously relying heavily on those markers to speed things up. Will be very interesting to see how the industry standardizes around this. Right now it's a bit of the wild west. Maybe people in ten years will look back at this post and think "what do you mean you judged people based on the code itself that's ridiculous"
- furyofantares 1y ago
  I think you're unfair to the heuristics people use in your framing here. You said "hence the point its author is trying to make must be true" and "hence it must be modeling the world with high fidelity". But it's more like "hence the author is likely competent and likely put in a reasonable effort." When those assumptions hold, putting in a very deep review is less likely to pay off. Maybe you are right that people have been too complacent to begin with, I don't know, but I don't think you've framed it fairly.

insane_dreamer 1y ago

> If someone uses an LLM and produces bug-free code, I'll trust them. Only because you already trust them to know that the code is indeed bug-free. Some cases are simple and straightforward -- this routine returns a desired value or it doesn't. Other situations are much more complex in anticipating the ways in which it might interact with other parts of the system, edge cases that are not obvious, etc. Writing code that is "bug free" in that situation requires the writer of the code to understand the implications of the code, and if the dev doesn't understand exactly what the code does because it was written by an LLM, then they won't be able to understand the implications of the code. It then falls to the reviewer to understand the implications of the code -- increasing their workload. That was the premise.

legacynl 1y ago

> How is this different from when they were only using their brain to produce the code? If A = B, higher B means that A is higher too. Now it's A + Ai = B. Now Higher B doesn't necessarily mean higher A. Especially since the current state of Ai is pretty much stochastic, and sometimes is worse than nothing at all

JambalayaJimbo 1y ago

I have never been in a work environment in which I’ve been able to do more than rubber stamp PRs. Performing a deep review of each change is simply impossible with the expectations we were given.

stavros 1y ago
Interesting, I've never been in one where we didn't read PRs.

cipherhood 1y ago

[dead]