6 ms·

The copy-paste programmer will always be worse than the programmer who builds a mental model of the system. LLMs are just a faster and more wrong version of th

The copy-paste programmer will always be worse than the programmer who builds a mental model of the system. LLMs are just a faster and more wrong version of the copy-paste stackoverflow workflow it’s just now you don’t even need to ask the right question to find the answer. You have to teach students and new engineers to never commit a piece of code they don’t understand. If you stop at “I don’t know why this works” then you will never be able to get out of the famous multi hour debug loop that you get into with LLMs or similarly the multi day build debugging loop that everyone has been through. The real thing that LLMs do that is bad for learning is that you don’t need to ask it the right question to find your answer. This is good if you already know the subject but if you don’t know the subject you’re not getting that reinforcement in your short term memory and you will find things you learned through LLMs are not retained as long as if you did more of it yourself.

stonethrowaway 2y ago
If engineers are still taught engineering as a discipline then it doesn’t matter what tools they use to achieve their goals. If we are calling software developers who don’t understand how things work, and who can get away with not knowing how things work, engineers, then that’s a separate discussion of profession and professionalism we should be having. As it stands there’s nothing fundamentally rooted in software developers having to understand why or how things work, which is why people can and do use the tools to get whatever output they’re after. I don’t see anything wrong with this. If anyone does, then feel free to change the curriculum so students are graded and tested on knowing how and why things work the way they do. The pearl clutching is boring and tiresome. Where required we have people who have to be licensed to perform certain work. And if they fail to perform it at that level their license is taken away. And if anyone wants to do unlicensed work then they are held accountable and will not receive any insurance coverage due to a lack of license. Meaning, they can be criminally held liable. This is why some countries go to the extent of requiring a license to call yourself an engineer at all. So where engineering, actual engineering, is required, we already have protocols in place that ensure things aren’t done on a “trust me bro” level. But for everyone else, they’re not held accountable whatsoever, and there’s nothing wrong with using whatever tools you need or want to use, right or wrong. If I want to butt splice a connector, I’m probably fine. But if I want to wire in a 3 phase breaker on a commercial property, I’m either looking at getting it done by someone licensed, or I’m looking at jail time if things go south. And engineering or no different.
- RodgerTheGreat 2y ago
  In many parts of the world, it is illegal to call yourself an "engineer" without both appropriate certification/training and legal accountability for the work one signs off upon, as with lawyers, medical doctors, and so on. It's frankly ridiculous that software "engineers" are permitted the title without the responsibility in the US.
  - stonethrowaway 2y ago
    Yet my comment keeps getting upvoted and downvoted. I guess I’m either saying something controversial, which I don’t think I am since I am stating the obvious, or potentially the anti-AI crowd doesn’t like my tone. I’m not pro or against AI (I don’t have a dog in this race). Everything at your disposal is potentially a tool to use how you see fit, whether it be AI or a screwdriver.
  - djeastm 2y ago
    >as with lawyers, medical doctors, and so on. It's frankly ridiculous that software "engineers" are permitted the title without the responsibility in the US. It's because 1) most of us don't work on things that can get people jailed or killed and 2) the US leans towards not regulating language so much. But if it makes you more comfortable, just think of the term "software engineer" as tongue-in-cheek, like some people call janitors "sanitation engineers"
  - ahtihn 2y ago
    Why do you want to police language? Is any employer mislead by the responsibilities software engineers actually have because of the title?
    - kazinator 2y ago
      Indeed. We don't have a fit about deception when somebody who unclogs drains calls themselves Drain Doctor.
- faizshah 2y ago
  If your goal is just to get something working then go right ahead. But if your goal is to be learning and improving your process and not introducing any new issues and not introducing a new threat etc. then you’re better off not just stopping at “it works” but also figuring out why it works and if this is the right way to make it work. The idea that wanting to become better at using something is pearl clutching is frankly why everything has become so mediocre.
nyrikki 2y ago
It is a bit more complicated, as it can be harmful for experts also, and the more reliable it gets the more problematic it becomes. Humans suffer from automation bias and other cognitive biases. Anything that causes disengagement from a process can be challenging, especially with long term maintainability and architectural erosion, which is mostly what I actively search for to try and avoid complacency with these tools. But it takes active effort to avoid for all humans. IMHO, writing the actual code has always been less of an issue than focusing on domain needs, details, and maintainability. As distrusting automation is unfortunately one of the best methods of fighting automation bias I do try to balance encouraging junior individuals to use tools that boost productivity while making sure they still maintain ownership of the delivered product. If you use the red-green refactor method, avoiding generative tools for the test and refactor steps seems to work. But selling TDD in general can be challenging especially with Holmström's theorem, and the tendency of people to write implementation tests vs focusing on domain needs. It is a bit of a paradox that the better the tools become, the higher the risk is, but I would encourage people to try the above, just don't make the mistake of copying the prompts required to get from red to green as the domain tests, there is a serious risk of coupling to the prompts. We will see if this works for me long term, but I do think beginners manually refactoring with though could be an accelerator. But only with intentionally focusing on learning why over time.
- nonrandomstring 2y ago
  > It is a bit of a paradox that the better the tools become, the higher the risk is "C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off". -- Bjarne Stroustrup
- faizshah 2y ago
  I completely reject this way of thinking. I remember when I was starting out it was popular to say you learn less by using an IDE and you should just use a text editor because you never learn how the system works if you rely on a run button or debug button or a WYSIWYG editor. Well in modern software we stand on the shoulders of many many giants, you have to start somewhere. Some things you may never need to learn (like say learning git at a deep level when learning the concept of add, commit, rebase, pull, push, cherry pick and reset are enough even if you use a GUI to do it) and some thins you might invest in over time (like learning things about your OS so you can optimize performance). The way you use automation effectively is to automate the things you don’t want to learn about and work on the things you do want to learn about. If you’re a backend dev who wants to learn how to write an API in Actix go ahead and copy paste some ChatGPT code, you just need to learn the shape of the API and the language first. If you’re a rust dev who wants to learn how Actix works don’t just copy and paste the code, get ChatGPT to give you a tutorial and then your write your API and use the docs yourself.
baxtr 2y ago
I wonder if this is an elitist argument. AI empowers normal people to start building stuff. Of course it won’t be as elegant and it will be bad for learning. However these people would have never learned anything about coding in the first place. Are we senior dev people a bit like carriage riders that complain about anyone being allowed to drive a car?
- UncleMeat 2y ago
  My spouse is a university professor. A lot of her students cheat using AI. I am sure that they could be using AI as a learning mechanism, but they observably aren't. Now, the motivations for using AI to pass a class are different but I think that it is important to recognize that there is using AI to build something and learn and there is using AI to build something. Engineering is also the process of development and maintenance over time. While an AI tool might help you build something that functions, that's just the first step. I am sure that there are people who leverage AI in such a way that the build a thing and also ask it a lot of questions about why it is built in a certain way and seek to internalize that. I'd wager that this is a small minority.
  - KoolKat23 2y ago
    Back in school on occasion it was considered cheating to use a calculator, the purpose to encourage learning. It would be absurd in the work environment to ban the use of calculators, it's your responsibility as an employee to use it correctly. As you say the first step.
    - UncleMeat 2y ago
      I'm sure at some point that universities will figure out how to integrate AI into pedagogy in a way that works other than a blanket ban. It also doesn't surprise me that until people figure out effective strategies that they say "no chatgpt on your homework."
- lawn 2y ago
  Maybe the senior developers are just jaded having to maintain code that nobody, not even their authors, know how it's supposed to work?
Buttons840 2y ago
You suggest learning the mental model behind the system, but is there a mental model behind web technologies? I'm reminded of the Wat talk: https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat Is it worth learning the mental model behind this system? Or am I better off just shoveling LLM slop around until it mostly works?
- tambourine_man 2y ago
  Of course there is. The DOM stands for Document Object Model. CSS uses the box model. A lot of thought went behind all these standards. JavaScript is weird, but show me a language that doesn’t have its warts.
  - senko 2y ago
    All languages have their warts. In JavaScript, the warts have their language.
    - joshuakcockrell 2y ago
      JavaScript needs a Python 2->3 moment, but like 100x the damage
  - hansvm 2y ago
    > JavaScript is weird, but show me a language that doesn’t have its warts. False equivalence much? Languages have warts. JS is a wart with just enough tumorous growth factors to have gained sentience and started its plans toward world domination.
- faizshah 2y ago
  The modern software space is too complex for any one person to know everything. There’s no one mental model. Your expertise over time comes from learning multiple mental models. For example if you are a frontend developer doing typescript in React you could learn how React’s renderer works or how typescript’s type system works or how the browser’s event listeners work. Over time you accumulate this knowledge through the projects you work on and the things you debug in prod. Or you can purposefully learn it through projects and studying. We also build up mental models of the way our product and it’s dependencies work. The reason a coworker might appear to be 10x or 100x more productive than you is because they are able to predict things about the system and arrive at solution faster. Why are they able to do that? It’s not because they use vim or type at 200 wpm. It’s because they have a mental of the way the system works that might be more refined than your own.
sgustard 2y ago
Quite often I'm incorporating a new library into my source. Every new library involves a choice: do I just spend 15 minutes on the Quick Start guide (i.e. "copy-paste"), or a day reading detailed docs, or a week investigating the complete source code? All of those are tradeoffs between understanding and time to market. LLMs are another tool to help navigate that tradeoff, and for me they continue to improve as I get better at asking the right questions.
- LtWorf 2y ago
  If you spend less than 15 minutes before even deciding which library to include and if include it at all. You're probably doing it wrong.
  - smikhanov 2y ago
    No, that person is doing it right. That’s 15 minutes of your life you’ll never get back; no library is worth it.
    - faizshah 2y ago
      If your goal is “ship it” then you might be right. If your goal is “ship it, and don’t break anything else, and don’t cause any security issues in the future and don’t rot the codebase, and be able to explain why you did it that way and why you didn’t use X” then you’re probably wrong.
    - LtWorf 2y ago
      That applies to working in general.
    - LtWorf 2y ago
      The past weeks at work have been spent replacing libraries that were picked in less than 15 minutes. Was this cost effective for the company?
- travisgriggs 2y ago
  Or "do I even need a library really?" These libraries do what I need AND so many other things that I don't need. Am I just bandwagoning. For my very simple purposes, maybe my own "prefix(n)" method is better than a big ol' library. Or not. All hail the mashup.
rsynnott 2y ago
I always wonder how much damage Stackoverflow did to programmer education, actually. There’s a certain type of programmer who will search for what they want to do, paste the first Stackoverflow answer which looks vaguely related, then the second, and so forth. This is particularly visible when interviewing people. It is… not a terribly effective approach to programming.
- noufalibrahim 2y ago
  I'd qualify that (and the llm situation) with a level of abstraction. It's one thing to have the llm generate a function call for you where you don't remember all the parameters. That's a low enough abstraction where it serves as a turbo charged doc lookup. It's also probably okay to get a basic setup (toolchain etc. for an ecosystem you're unfamilar with(. But to have it solve entire problems for you especially when you're learning is a disaster.
- exe34 2y ago
  my workflow with stackoverflow is to try to get working code that does the minimum of what I'm trying to do, and only try to understand it after it works as I want it to. otherwise there's an infinite amount of code out there that doesn't work (because of version incompatibility, plain wrong code, etc) and I ran out of patience long ago. if it doesn't run, I don't want to understand it.
  - faizshah 2y ago
    This is in my opinion the right way to use it. You can use stackoverflow or ChatGPT to get to “It works!” But don’t stop there, stop at “It works, and I know why it works and I think this is the best way to do it.” If you just stop at “It works!” You didn’t learn anything and might be unknowingly making new problems.
  - username135 2y ago
    My general philosophy as well.
- underlipton 2y ago
valval 2y ago
I wouldn’t be super worried about this, since there will always be pioneers and followers and there’s no proof to suggest the ratio of those hasn’t remained somewhat constant throughout history.

stonethrowaway 2y ago

If engineers are still taught engineering as a discipline then it doesn’t matter what tools they use to achieve their goals. If we are calling software developers who don’t understand how things work, and who can get away with not knowing how things work, engineers, then that’s a separate discussion of profession and professionalism we should be having. As it stands there’s nothing fundamentally rooted in software developers having to understand why or how things work, which is why people can and do use the tools to get whatever output they’re after. I don’t see anything wrong with this. If anyone does, then feel free to change the curriculum so students are graded and tested on knowing how and why things work the way they do. The pearl clutching is boring and tiresome. Where required we have people who have to be licensed to perform certain work. And if they fail to perform it at that level their license is taken away. And if anyone wants to do unlicensed work then they are held accountable and will not receive any insurance coverage due to a lack of license. Meaning, they can be criminally held liable. This is why some countries go to the extent of requiring a license to call yourself an engineer at all. So where engineering, actual engineering, is required, we already have protocols in place that ensure things aren’t done on a “trust me bro” level. But for everyone else, they’re not held accountable whatsoever, and there’s nothing wrong with using whatever tools you need or want to use, right or wrong. If I want to butt splice a connector, I’m probably fine. But if I want to wire in a 3 phase breaker on a commercial property, I’m either looking at getting it done by someone licensed, or I’m looking at jail time if things go south. And engineering or no different.

RodgerTheGreat 2y ago
In many parts of the world, it is illegal to call yourself an "engineer" without both appropriate certification/training and legal accountability for the work one signs off upon, as with lawyers, medical doctors, and so on. It's frankly ridiculous that software "engineers" are permitted the title without the responsibility in the US.
- stonethrowaway 2y ago
  Yet my comment keeps getting upvoted and downvoted. I guess I’m either saying something controversial, which I don’t think I am since I am stating the obvious, or potentially the anti-AI crowd doesn’t like my tone. I’m not pro or against AI (I don’t have a dog in this race). Everything at your disposal is potentially a tool to use how you see fit, whether it be AI or a screwdriver.
- djeastm 2y ago
  >as with lawyers, medical doctors, and so on. It's frankly ridiculous that software "engineers" are permitted the title without the responsibility in the US. It's because 1) most of us don't work on things that can get people jailed or killed and 2) the US leans towards not regulating language so much. But if it makes you more comfortable, just think of the term "software engineer" as tongue-in-cheek, like some people call janitors "sanitation engineers"
- ahtihn 2y ago
  Why do you want to police language? Is any employer mislead by the responsibilities software engineers actually have because of the title?
  - kazinator 2y ago
    Indeed. We don't have a fit about deception when somebody who unclogs drains calls themselves Drain Doctor.
faizshah 2y ago
If your goal is just to get something working then go right ahead. But if your goal is to be learning and improving your process and not introducing any new issues and not introducing a new threat etc. then you’re better off not just stopping at “it works” but also figuring out why it works and if this is the right way to make it work. The idea that wanting to become better at using something is pearl clutching is frankly why everything has become so mediocre.

nyrikki 2y ago

It is a bit more complicated, as it can be harmful for experts also, and the more reliable it gets the more problematic it becomes. Humans suffer from automation bias and other cognitive biases. Anything that causes disengagement from a process can be challenging, especially with long term maintainability and architectural erosion, which is mostly what I actively search for to try and avoid complacency with these tools. But it takes active effort to avoid for all humans. IMHO, writing the actual code has always been less of an issue than focusing on domain needs, details, and maintainability. As distrusting automation is unfortunately one of the best methods of fighting automation bias I do try to balance encouraging junior individuals to use tools that boost productivity while making sure they still maintain ownership of the delivered product. If you use the red-green refactor method, avoiding generative tools for the test and refactor steps seems to work. But selling TDD in general can be challenging especially with Holmström's theorem, and the tendency of people to write implementation tests vs focusing on domain needs. It is a bit of a paradox that the better the tools become, the higher the risk is, but I would encourage people to try the above, just don't make the mistake of copying the prompts required to get from red to green as the domain tests, there is a serious risk of coupling to the prompts. We will see if this works for me long term, but I do think beginners manually refactoring with though could be an accelerator. But only with intentionally focusing on learning why over time.

nonrandomstring 2y ago
> It is a bit of a paradox that the better the tools become, the higher the risk is "C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off". -- Bjarne Stroustrup
faizshah 2y ago
I completely reject this way of thinking. I remember when I was starting out it was popular to say you learn less by using an IDE and you should just use a text editor because you never learn how the system works if you rely on a run button or debug button or a WYSIWYG editor. Well in modern software we stand on the shoulders of many many giants, you have to start somewhere. Some things you may never need to learn (like say learning git at a deep level when learning the concept of add, commit, rebase, pull, push, cherry pick and reset are enough even if you use a GUI to do it) and some thins you might invest in over time (like learning things about your OS so you can optimize performance). The way you use automation effectively is to automate the things you don’t want to learn about and work on the things you do want to learn about. If you’re a backend dev who wants to learn how to write an API in Actix go ahead and copy paste some ChatGPT code, you just need to learn the shape of the API and the language first. If you’re a rust dev who wants to learn how Actix works don’t just copy and paste the code, get ChatGPT to give you a tutorial and then your write your API and use the docs yourself.

baxtr 2y ago

I wonder if this is an elitist argument. AI empowers normal people to start building stuff. Of course it won’t be as elegant and it will be bad for learning. However these people would have never learned anything about coding in the first place. Are we senior dev people a bit like carriage riders that complain about anyone being allowed to drive a car?

UncleMeat 2y ago
My spouse is a university professor. A lot of her students cheat using AI. I am sure that they could be using AI as a learning mechanism, but they observably aren't. Now, the motivations for using AI to pass a class are different but I think that it is important to recognize that there is using AI to build something and learn and there is using AI to build something. Engineering is also the process of development and maintenance over time. While an AI tool might help you build something that functions, that's just the first step. I am sure that there are people who leverage AI in such a way that the build a thing and also ask it a lot of questions about why it is built in a certain way and seek to internalize that. I'd wager that this is a small minority.
- KoolKat23 2y ago
  Back in school on occasion it was considered cheating to use a calculator, the purpose to encourage learning. It would be absurd in the work environment to ban the use of calculators, it's your responsibility as an employee to use it correctly. As you say the first step.
  - UncleMeat 2y ago
    I'm sure at some point that universities will figure out how to integrate AI into pedagogy in a way that works other than a blanket ban. It also doesn't surprise me that until people figure out effective strategies that they say "no chatgpt on your homework."
lawn 2y ago
Maybe the senior developers are just jaded having to maintain code that nobody, not even their authors, know how it's supposed to work?

Buttons840 2y ago

You suggest learning the mental model behind the system, but is there a mental model behind web technologies? I'm reminded of the Wat talk: https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat Is it worth learning the mental model behind this system? Or am I better off just shoveling LLM slop around until it mostly works?

tambourine_man 2y ago
Of course there is. The DOM stands for Document Object Model. CSS uses the box model. A lot of thought went behind all these standards. JavaScript is weird, but show me a language that doesn’t have its warts.
- senko 2y ago
  All languages have their warts. In JavaScript, the warts have their language.
  - joshuakcockrell 2y ago
    JavaScript needs a Python 2->3 moment, but like 100x the damage
- hansvm 2y ago
  > JavaScript is weird, but show me a language that doesn’t have its warts. False equivalence much? Languages have warts. JS is a wart with just enough tumorous growth factors to have gained sentience and started its plans toward world domination.
faizshah 2y ago
The modern software space is too complex for any one person to know everything. There’s no one mental model. Your expertise over time comes from learning multiple mental models. For example if you are a frontend developer doing typescript in React you could learn how React’s renderer works or how typescript’s type system works or how the browser’s event listeners work. Over time you accumulate this knowledge through the projects you work on and the things you debug in prod. Or you can purposefully learn it through projects and studying. We also build up mental models of the way our product and it’s dependencies work. The reason a coworker might appear to be 10x or 100x more productive than you is because they are able to predict things about the system and arrive at solution faster. Why are they able to do that? It’s not because they use vim or type at 200 wpm. It’s because they have a mental of the way the system works that might be more refined than your own.

sgustard 2y ago

Quite often I'm incorporating a new library into my source. Every new library involves a choice: do I just spend 15 minutes on the Quick Start guide (i.e. "copy-paste"), or a day reading detailed docs, or a week investigating the complete source code? All of those are tradeoffs between understanding and time to market. LLMs are another tool to help navigate that tradeoff, and for me they continue to improve as I get better at asking the right questions.

LtWorf 2y ago
If you spend less than 15 minutes before even deciding which library to include and if include it at all. You're probably doing it wrong.
- smikhanov 2y ago
  No, that person is doing it right. That’s 15 minutes of your life you’ll never get back; no library is worth it.
  - faizshah 2y ago
    If your goal is “ship it” then you might be right. If your goal is “ship it, and don’t break anything else, and don’t cause any security issues in the future and don’t rot the codebase, and be able to explain why you did it that way and why you didn’t use X” then you’re probably wrong.
  - LtWorf 2y ago
    That applies to working in general.
  - LtWorf 2y ago
    The past weeks at work have been spent replacing libraries that were picked in less than 15 minutes. Was this cost effective for the company?
travisgriggs 2y ago
Or "do I even need a library really?" These libraries do what I need AND so many other things that I don't need. Am I just bandwagoning. For my very simple purposes, maybe my own "prefix(n)" method is better than a big ol' library. Or not. All hail the mashup.

rsynnott 2y ago

I always wonder how much damage Stackoverflow did to programmer education, actually. There’s a certain type of programmer who will search for what they want to do, paste the first Stackoverflow answer which looks vaguely related, then the second, and so forth. This is particularly visible when interviewing people. It is… not a terribly effective approach to programming.

noufalibrahim 2y ago
I'd qualify that (and the llm situation) with a level of abstraction. It's one thing to have the llm generate a function call for you where you don't remember all the parameters. That's a low enough abstraction where it serves as a turbo charged doc lookup. It's also probably okay to get a basic setup (toolchain etc. for an ecosystem you're unfamilar with(. But to have it solve entire problems for you especially when you're learning is a disaster.
exe34 2y ago
my workflow with stackoverflow is to try to get working code that does the minimum of what I'm trying to do, and only try to understand it after it works as I want it to. otherwise there's an infinite amount of code out there that doesn't work (because of version incompatibility, plain wrong code, etc) and I ran out of patience long ago. if it doesn't run, I don't want to understand it.
- faizshah 2y ago
  This is in my opinion the right way to use it. You can use stackoverflow or ChatGPT to get to “It works!” But don’t stop there, stop at “It works, and I know why it works and I think this is the best way to do it.” If you just stop at “It works!” You didn’t learn anything and might be unknowingly making new problems.
- username135 2y ago
  My general philosophy as well.
underlipton 2y ago

valval 2y ago

I wouldn’t be super worried about this, since there will always be pioneers and followers and there’s no proof to suggest the ratio of those hasn’t remained somewhat constant throughout history.