5 ms·

Is "what has been my experience" not implied in what I am still waiting for — "someone using an LLM to produce high quality code in the field of their expertise

Is "what has been my experience" not implied in what I am still waiting for — "someone using an LLM to produce high quality code in the field of their expertise in less time than without an LLM"? So, observing a couple of my colleagues (I am an engineering manager, but have switched back and forth between management and IC roles for the last ~20 years), I've seen them either produce crap, or spend so much time tuning the prompts that it would have been faster to do it without an LLM. They mostly used Github Copilot or ChatGPT (most recent versions as of last few months ago). I am also keeping out a keen eye for any examples of this (on HN in particular), but it usually turns out things like https://news.ycombinator.com/item?id=43573755 https://news.ycombinator.com/item?id=43573755 Again, I am not saying it's not being done, but I have struggled to find someone who would demonstrate it happen in a convincing enough fashion — I am really trying to imagine how I would best incorporate this into my daily non-work programming activities, so I'd love to see a few examples of someone using it effectively.

tptacek 2y ago
LLMs don't make bad programmers into good programmers. If your team is comfortable merging "crap", you have deeper problems.
- necovek 2y ago
  If you don't want to engage in an honest discussion, please refrain from making assumptions: nobody mentioned "merging crap". I stopped clearly at guiding an LLM to make a code change, which is at best, a pull request.
  - tptacek 2y ago
    How is that better? As a team lead, what would you think of a team member who consistently generated "crap" pull requests? You see the same thing in every argument with LLM skeptics. 'The code is bad. You don't even know what the code is doing." This is obviously false. A professional reads the code they commit and push. A professional doesn't push code they know to be bad.
    - necovek 2y ago
      "Let's identify specific, clear areas for improvement, and if they are not able to improve, let's fire them": it's as simple as that. Teaching a human with motivation, potential and desire to learn is both easier, and more rewarding (for most humans), than attempting to teach LLM to write good code every time — humans tend to value their personal experiences more, whereas LLM relies more on the training corpus. So when I've seen people massage LLM output to be decent or excellent, it took them more time than it would have taken for them to write it from scratch without an LLM. Which makes LLMs mostly a curiosity, and not a productivity booster. Can it get there? I hope it can, because that would be amazing.
      - tptacek 2y ago
        None of this responds to what I just wrote. Can you engage with the question I asked directly? Thanks!
        necovek 2y ago
        You asked: > As a team lead, what would you think of a team member who consistently generated "crap" pull requests? This is directly answered with my first paragraph: that's exactly what I would think of them, and how I would act on it. Your first question was: > How is that better? In the second paragraph, I explained why it's better to do a code review for a crappy pull request that's human-produced vs LLM-generated: it is easier, faster, and more psychologically rewarding. If you are talking about a case where an inexperienced human uses LLM to start off with a crappy code change, but then adapts the output during the review process, and potentially learns through it (though research confirms people learn better when they produce mistakes themselves) — they still won't be able to use LLM to produce comparable code the next time, so they'll have to do the review and improve it by hand before putting it up for review by somebody else, thus negating any productivity gain (which was the original premise), and likely reducing the learning potential. If there was a question I misinterpreted, please enlighten me. Thanks! :)
        2y ago

tptacek 2y ago

LLMs don't make bad programmers into good programmers. If your team is comfortable merging "crap", you have deeper problems.

necovek 2y ago
If you don't want to engage in an honest discussion, please refrain from making assumptions: nobody mentioned "merging crap". I stopped clearly at guiding an LLM to make a code change, which is at best, a pull request.
- tptacek 2y ago
  How is that better? As a team lead, what would you think of a team member who consistently generated "crap" pull requests? You see the same thing in every argument with LLM skeptics. 'The code is bad. You don't even know what the code is doing." This is obviously false. A professional reads the code they commit and push. A professional doesn't push code they know to be bad.
  - necovek 2y ago
    "Let's identify specific, clear areas for improvement, and if they are not able to improve, let's fire them": it's as simple as that. Teaching a human with motivation, potential and desire to learn is both easier, and more rewarding (for most humans), than attempting to teach LLM to write good code every time — humans tend to value their personal experiences more, whereas LLM relies more on the training corpus. So when I've seen people massage LLM output to be decent or excellent, it took them more time than it would have taken for them to write it from scratch without an LLM. Which makes LLMs mostly a curiosity, and not a productivity booster. Can it get there? I hope it can, because that would be amazing.
    - tptacek 2y ago
      None of this responds to what I just wrote. Can you engage with the question I asked directly? Thanks!
      - necovek 2y ago
        You asked: > As a team lead, what would you think of a team member who consistently generated "crap" pull requests? This is directly answered with my first paragraph: that's exactly what I would think of them, and how I would act on it. Your first question was: > How is that better? In the second paragraph, I explained why it's better to do a code review for a crappy pull request that's human-produced vs LLM-generated: it is easier, faster, and more psychologically rewarding. If you are talking about a case where an inexperienced human uses LLM to start off with a crappy code change, but then adapts the output during the review process, and potentially learns through it (though research confirms people learn better when they produce mistakes themselves) — they still won't be able to use LLM to produce comparable code the next time, so they'll have to do the review and improve it by hand before putting it up for review by somebody else, thus negating any productivity gain (which was the original premise), and likely reducing the learning potential. If there was a question I misinterpreted, please enlighten me. Thanks! :)
        2y ago