4 ms·

I think they summed it up well in the section "Why do we write, anyway?" — they nowhere claimed it was vital for students' success in a robotic class. On the co

I think they summed it up well in the section "Why do we write, anyway?" — they nowhere claimed it was vital for students' success in a robotic class. On the contrary as they title a subsection there with "If it’s worth doing, it’s worth doing *badly*" (emphasis mine) — so what they are looking for is to peer into the author's mind and their original thoughts. The implication there is that this is acceptable to pass a robotics class, and potentially this gives them more information about students' comprehension to further improve their instruction and teaching ("...that they have some kind of internal understanding to share"). On that second point, I have yet to see someone demonstrate a "smart, professional programmer use LLMs" in a way where it produces high quality output in their area of expertise, while improving their efficiency and thus saving time for them (compared to them just using a good, old IDE)!

tptacek 2y ago
Who have you asked? What has been your experience observing professionals using LLMs?
- necovek 2y ago
  Is "what has been my experience" not implied in what I am still waiting for — "someone using an LLM to produce high quality code in the field of their expertise in less time than without an LLM"? So, observing a couple of my colleagues (I am an engineering manager, but have switched back and forth between management and IC roles for the last ~20 years), I've seen them either produce crap, or spend so much time tuning the prompts that it would have been faster to do it without an LLM. They mostly used Github Copilot or ChatGPT (most recent versions as of last few months ago). I am also keeping out a keen eye for any examples of this (on HN in particular), but it usually turns out things like https://news.ycombinator.com/item?id=43573755 https://news.ycombinator.com/item?id=43573755 Again, I am not saying it's not being done, but I have struggled to find someone who would demonstrate it happen in a convincing enough fashion — I am really trying to imagine how I would best incorporate this into my daily non-work programming activities, so I'd love to see a few examples of someone using it effectively.
  - tptacek 2y ago
    LLMs don't make bad programmers into good programmers. If your team is comfortable merging "crap", you have deeper problems.
    - necovek 2y ago
      If you don't want to engage in an honest discussion, please refrain from making assumptions: nobody mentioned "merging crap". I stopped clearly at guiding an LLM to make a code change, which is at best, a pull request.
      - tptacek 2y ago
        How is that better? As a team lead, what would you think of a team member who consistently generated "crap" pull requests? You see the same thing in every argument with LLM skeptics. 'The code is bad. You don't even know what the code is doing." This is obviously false. A professional reads the code they commit and push. A professional doesn't push code they know to be bad.
        necovek 2y ago
        "Let's identify specific, clear areas for improvement, and if they are not able to improve, let's fire them": it's as simple as that. Teaching a human with motivation, potential and desire to learn is both easier, and more rewarding (for most humans), than attempting to teach LLM to write good code every time — humans tend to value their personal experiences more, whereas LLM relies more on the training corpus. So when I've seen people massage LLM output to be decent or excellent, it took them more time than it would have taken for them to write it from scratch without an LLM. Which makes LLMs mostly a curiosity, and not a productivity booster. Can it get there? I hope it can, because that would be amazing.
        tptacek 2y ago
        None of this responds to what I just wrote. Can you engage with the question I asked directly? Thanks!
dcre 2y ago
A couple of examples from influential open source developers: Adam Wathan (Tailwind) agreeing with Mitchell Hashimoto that LLMs are making them more productive. "in their area of expertise" is not obvious from this post alone, but I am pretty confident from the way they talk about it that they are not exclusively using LLMs on side projects where they're inexpert. https://x.com/adamwathan/status/1911845073286803923 https://x.com/adamwathan/status/1911845073286803923 Armin Ronacher also talks about using LLMs quite a bit, but I don't have as good of an example from his tweets of him straightforwardly saying "yes, they are useful to me!"
- necovek 2y ago
  "Saying" vs "demonstrating": I am careful with the words I am using. It's not just nitpicking: maybe it's only their perception they are more productive? Or maybe they are. But saying it is not convincing enough for me.
  - dcre 2y ago
    That's a good point, though it's worth thinking about what would count as a demonstration. Pointing to a PR that an LLM helped with is not enough because you don't know how long it would have taken without the use of LLMs. The counterfactual never really exists. What is the hard evidence vim or emacs makes people more productive? Using git over svn? Using search engines? Over time, we expect people to use the things that make them more productive.
    - necovek 1y ago
      I would assume honesty, so someone showing with a blog post their incremental progress, prompting or code assistant use, rough time spent on each iteration... would be a great start. I am not looking for formal study level of trust (though even that is frequently debatable), but multiple accounts of this where there is clear quality of output and significant (estimated) time savings would be wonderful. I don't need much convincing that for many engineers, an LLM can bring incremental speed up (10-20%), though I think that really depends on the personality (eg. do you prefer to fix stuff not created by you or write it nicely from the start?).

tptacek 2y ago

Who have you asked? What has been your experience observing professionals using LLMs?

necovek 2y ago
Is "what has been my experience" not implied in what I am still waiting for — "someone using an LLM to produce high quality code in the field of their expertise in less time than without an LLM"? So, observing a couple of my colleagues (I am an engineering manager, but have switched back and forth between management and IC roles for the last ~20 years), I've seen them either produce crap, or spend so much time tuning the prompts that it would have been faster to do it without an LLM. They mostly used Github Copilot or ChatGPT (most recent versions as of last few months ago). I am also keeping out a keen eye for any examples of this (on HN in particular), but it usually turns out things like https://news.ycombinator.com/item?id=43573755 https://news.ycombinator.com/item?id=43573755 Again, I am not saying it's not being done, but I have struggled to find someone who would demonstrate it happen in a convincing enough fashion — I am really trying to imagine how I would best incorporate this into my daily non-work programming activities, so I'd love to see a few examples of someone using it effectively.
- tptacek 2y ago
  LLMs don't make bad programmers into good programmers. If your team is comfortable merging "crap", you have deeper problems.
  - necovek 2y ago
    If you don't want to engage in an honest discussion, please refrain from making assumptions: nobody mentioned "merging crap". I stopped clearly at guiding an LLM to make a code change, which is at best, a pull request.
    - tptacek 2y ago
      How is that better? As a team lead, what would you think of a team member who consistently generated "crap" pull requests? You see the same thing in every argument with LLM skeptics. 'The code is bad. You don't even know what the code is doing." This is obviously false. A professional reads the code they commit and push. A professional doesn't push code they know to be bad.
      - necovek 2y ago
        "Let's identify specific, clear areas for improvement, and if they are not able to improve, let's fire them": it's as simple as that. Teaching a human with motivation, potential and desire to learn is both easier, and more rewarding (for most humans), than attempting to teach LLM to write good code every time — humans tend to value their personal experiences more, whereas LLM relies more on the training corpus. So when I've seen people massage LLM output to be decent or excellent, it took them more time than it would have taken for them to write it from scratch without an LLM. Which makes LLMs mostly a curiosity, and not a productivity booster. Can it get there? I hope it can, because that would be amazing.
        tptacek 2y ago
        None of this responds to what I just wrote. Can you engage with the question I asked directly? Thanks!

dcre 2y ago

A couple of examples from influential open source developers: Adam Wathan (Tailwind) agreeing with Mitchell Hashimoto that LLMs are making them more productive. "in their area of expertise" is not obvious from this post alone, but I am pretty confident from the way they talk about it that they are not exclusively using LLMs on side projects where they're inexpert. https://x.com/adamwathan/status/1911845073286803923 https://x.com/adamwathan/status/1911845073286803923 Armin Ronacher also talks about using LLMs quite a bit, but I don't have as good of an example from his tweets of him straightforwardly saying "yes, they are useful to me!"

necovek 2y ago
"Saying" vs "demonstrating": I am careful with the words I am using. It's not just nitpicking: maybe it's only their perception they are more productive? Or maybe they are. But saying it is not convincing enough for me.
- dcre 2y ago
  That's a good point, though it's worth thinking about what would count as a demonstration. Pointing to a PR that an LLM helped with is not enough because you don't know how long it would have taken without the use of LLMs. The counterfactual never really exists. What is the hard evidence vim or emacs makes people more productive? Using git over svn? Using search engines? Over time, we expect people to use the things that make them more productive.
  - necovek 1y ago
    I would assume honesty, so someone showing with a blog post their incremental progress, prompting or code assistant use, rough time spent on each iteration... would be a great start. I am not looking for formal study level of trust (though even that is frequently debatable), but multiple accounts of this where there is clear quality of output and significant (estimated) time savings would be wonderful. I don't need much convincing that for many engineers, an LLM can bring incremental speed up (10-20%), though I think that really depends on the personality (eg. do you prefer to fix stuff not created by you or write it nicely from the start?).