5 ms·

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors. My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they've still not (fully) understood. The process of experimenting, learning and understanding is then largely replaced by the LLM, and the real skill becomes applying prompt tweaks until it looks like stuff works.

arealaccount 2y ago
Why I recommend something like CoPilot over ChatGPT to juniors wanting to use AI. At least Copilot you’re still more or less driving, whereas ChatGPT you’re more the passenger and not growing intuition.
DanHulton 2y ago
I was really hoping this study would be exploring that. Really, it's examining short-term productivity gains, ignoring long-term tech debt that can occur, and _completely_ ignoring effects on the growth of the software developers themselves. I share your hunch, though I would go so far as to call it an informed, strong opinion. I think we're going to pay the price in this industry in a few years, where the pipeline of "clueful junior software developers" is gonna dry way up, replaced by a firehose of "AI-reliant junior software developers", and the distance between those two categories is a GULF. (And of course, it has a knock-on effect on the number of clueful intermediate software developers, and clueful senior software developers, etc...)
diob 2y ago
I think it can be abused like anything else (copy paste from stack overflow until it works). But for folks who are going to be successful with or without it, it's a godsend in terms of being able to essentially ask stack overflow questions and get immediate non judgemental answers. Maybe not correct all the time, but that was true with stack overflow as well. So as always, it comes back to the individual.
throwaway314155 2y ago
My experience has been that indeed, it is detrimental to juniors. But unlike your take, it is largely a boon to experienced developers. That you suggest "tedium" is involved for more senior developers suggests to me that you haven't given the tooling a fair chance or work with a relatively obscure technology/language.
- layer8 2y ago
  I think you’ve misunderstood the GP. They are saying AI is useful to seniors for tasks that would otherwise be tedious, but doing those tedious tasks by hand would be formative for juniors, and it is detrimental to their growth when they do them using AI.
  - throwaway314155 2y ago
    Ah yeah you're right. In my defense, that sentence reads ambiguously without a couple of re-reads. There's a (not actually) implied double negative in there, or something, which threw me off. Thanks for pointing it out with words instead of downvotes.
    - keybored 2y ago
      [flagged]
      - throwaway314155 2y ago
        [flagged]
Workaccount2 2y ago
With the progress LLM's having been making in the last two years, is it actually a bad bet to not want to really get into it? How many contemporary developers have no idea how to write machine code, when 50 years ago it was basically mandatory if you wanted to be able to write anything? Are LLM's just going to become another abstraction crutch turned abstraction solid pillar?
- jprete 2y ago
  LLMs aren't an abstraction, even a very leaky one, so the analogies with compilers and the like really fall flat for me.
- arexxbifs 2y ago
  Abstraction is beneficial and profitable up to a certain point, after which upkeep gets too hard or expensive, and knowledge dwindles into a competency crisis - for various reasons. I'm not saying we are at that point yet, but it feels like we're closing in on it (and not just in software development). 50 years ago isn't even 50 years ago anymore, if you catch my drift: In 1974, the real king of the hill was COBOL - a very straight-forward abstraction. I'm seeing a lot of confusion and frustration from beginner programmers when it comes to abstraction, because a lot of abstractions in use today just incur other kinds of complexity. At a glance, React for example can seem deceptively easy, but in truth it requires understanding of a lot of advanced concepts. And sure, a little knowledge can go a long way in E.G. web development, but to really write robust, performant code you have to know a lot about the browser it runs in, not unlike how great programmers of yesteryear had entire 8-bit machines mapped out in their heads. Considering this, I'm not convinced the LLM crutch will ever solidify into a pillar of understanding and maintainable competence.
  - skydhash 2y ago
    And it really helps if you have a global view across the abstraction stack, even if you don't dive in the details of the implementation. I still think that having some computer organization/OS architecture knowledge would be great for developers, at least to know that memory is not free, even though we have GBs of it and that having an internet connection is not an integral part of the computer like the power supply.
tensor 2y ago
I think it really depends on the users. The same people who would just paste stackoverflow code until it seems to work and call it a day will abuse LLMs. However, those of us who like to know everything about the code we write will likely research anything an LLM spits out that we don't know about. Well, at least that's how I use them. And to throw a counter to your hypothesis, I find that sometimes the LLM will use functions or library components that I didn't know of, which actually saves me a lot of time when learning a new language or toolkit. So for me, it actually accelerates learning rather than retarding it.
pphysch 2y ago
It all depends of one is using it scientifically, i.e. having a hypothesis for the generated code before it exists, so that you can evaluate it. When used scientifically, coding copilots boost productivity AND skills.
saintradon 2y ago
Like any other tool, it depends on how you use it.
JackMorgan 2y ago
For myself it's an incredible tool for learning. I learn both broader and deeper using chat tools. If anything it gives me a great "sounding board" for exploring a subject and finding additional resources. E.g last night I setup my first Linux raid. A task that isn't too hard, but following a tutorial or just "reading the docs" isn't particularly helpful given it takes a few different tools (mount, umount, fstab, blkid, mdadm, fdisk, lsblk, mkfs) and along the way things might not follow the exact steps from a guide. I asked dozens of questions about each tool and step, where previously I would have just "copy paste and prayed". Two nights ago I was similarly able to fully recover all my data from a failed ssd also using chatgpt to guide my learning along the way. It was really cool to tackle a completely new skill having a "guide" even if it's wrong 20% of the time, that's way better than the average on the open Internet. For someone who loves learning, it feels like thousand league boots compared to just endlessly sifting through internet crap. Of course everything it says is suspect, just like everything else on the Internet, but boy it cuts out a lot of the hassle.
- iLoveOncall 2y ago
  Okay but this is not programming, more about an enhanced tutorial. This isn't at all what the original commenter is talking about.
- manmal 2y ago
  You’ve given two examples for “broad”, but none for “deep”. I’ve also used LLMs for setting up my homelab, and they were really helpful since I was basically at beginner level in most linux admin topics (still am). But trying to eg setup automatic snapshot replications for my zfs pool had me go back to reading blog posts, as ChatGPT just couldn’t provide a solution that worked for me.
  - Spivak 2y ago
    I think one of the catch-22s of LLMs is that using it as a fancy search index (which is the dev assistant use-case) is that the information it surfaces is hugely dependent on what words you use and it matches energy. If you don't know the words you'll get very beginner oriented content and you can get it to surface deeper knowledge iff you know the shibboleths, but it's annoying. One dumb trick that's been unreasonably useful is just copy-pasting barely-related source code, but at the level you're trying to understand and then just asking an unrelated question— just yank that search vector way over to the region where you think the good information lives.

arealaccount 2y ago

Why I recommend something like CoPilot over ChatGPT to juniors wanting to use AI. At least Copilot you’re still more or less driving, whereas ChatGPT you’re more the passenger and not growing intuition.

DanHulton 2y ago

I was really hoping this study would be exploring that. Really, it's examining short-term productivity gains, ignoring long-term tech debt that can occur, and _completely_ ignoring effects on the growth of the software developers themselves. I share your hunch, though I would go so far as to call it an informed, strong opinion. I think we're going to pay the price in this industry in a few years, where the pipeline of "clueful junior software developers" is gonna dry way up, replaced by a firehose of "AI-reliant junior software developers", and the distance between those two categories is a GULF. (And of course, it has a knock-on effect on the number of clueful intermediate software developers, and clueful senior software developers, etc...)

diob 2y ago

I think it can be abused like anything else (copy paste from stack overflow until it works). But for folks who are going to be successful with or without it, it's a godsend in terms of being able to essentially ask stack overflow questions and get immediate non judgemental answers. Maybe not correct all the time, but that was true with stack overflow as well. So as always, it comes back to the individual.

throwaway314155 2y ago

My experience has been that indeed, it is detrimental to juniors. But unlike your take, it is largely a boon to experienced developers. That you suggest "tedium" is involved for more senior developers suggests to me that you haven't given the tooling a fair chance or work with a relatively obscure technology/language.

layer8 2y ago
I think you’ve misunderstood the GP. They are saying AI is useful to seniors for tasks that would otherwise be tedious, but doing those tedious tasks by hand would be formative for juniors, and it is detrimental to their growth when they do them using AI.
- throwaway314155 2y ago
  Ah yeah you're right. In my defense, that sentence reads ambiguously without a couple of re-reads. There's a (not actually) implied double negative in there, or something, which threw me off. Thanks for pointing it out with words instead of downvotes.
  - keybored 2y ago
    [flagged]
    - throwaway314155 2y ago
      [flagged]

Workaccount2 2y ago

With the progress LLM's having been making in the last two years, is it actually a bad bet to not want to really get into it? How many contemporary developers have no idea how to write machine code, when 50 years ago it was basically mandatory if you wanted to be able to write anything? Are LLM's just going to become another abstraction crutch turned abstraction solid pillar?

jprete 2y ago
LLMs aren't an abstraction, even a very leaky one, so the analogies with compilers and the like really fall flat for me.
arexxbifs 2y ago
Abstraction is beneficial and profitable up to a certain point, after which upkeep gets too hard or expensive, and knowledge dwindles into a competency crisis - for various reasons. I'm not saying we are at that point yet, but it feels like we're closing in on it (and not just in software development). 50 years ago isn't even 50 years ago anymore, if you catch my drift: In 1974, the real king of the hill was COBOL - a very straight-forward abstraction. I'm seeing a lot of confusion and frustration from beginner programmers when it comes to abstraction, because a lot of abstractions in use today just incur other kinds of complexity. At a glance, React for example can seem deceptively easy, but in truth it requires understanding of a lot of advanced concepts. And sure, a little knowledge can go a long way in E.G. web development, but to really write robust, performant code you have to know a lot about the browser it runs in, not unlike how great programmers of yesteryear had entire 8-bit machines mapped out in their heads. Considering this, I'm not convinced the LLM crutch will ever solidify into a pillar of understanding and maintainable competence.
- skydhash 2y ago
  And it really helps if you have a global view across the abstraction stack, even if you don't dive in the details of the implementation. I still think that having some computer organization/OS architecture knowledge would be great for developers, at least to know that memory is not free, even though we have GBs of it and that having an internet connection is not an integral part of the computer like the power supply.

tensor 2y ago

I think it really depends on the users. The same people who would just paste stackoverflow code until it seems to work and call it a day will abuse LLMs. However, those of us who like to know everything about the code we write will likely research anything an LLM spits out that we don't know about. Well, at least that's how I use them. And to throw a counter to your hypothesis, I find that sometimes the LLM will use functions or library components that I didn't know of, which actually saves me a lot of time when learning a new language or toolkit. So for me, it actually accelerates learning rather than retarding it.

pphysch 2y ago

It all depends of one is using it scientifically, i.e. having a hypothesis for the generated code before it exists, so that you can evaluate it. When used scientifically, coding copilots boost productivity AND skills.

saintradon 2y ago

Like any other tool, it depends on how you use it.

JackMorgan 2y ago

For myself it's an incredible tool for learning. I learn both broader and deeper using chat tools. If anything it gives me a great "sounding board" for exploring a subject and finding additional resources. E.g last night I setup my first Linux raid. A task that isn't too hard, but following a tutorial or just "reading the docs" isn't particularly helpful given it takes a few different tools (mount, umount, fstab, blkid, mdadm, fdisk, lsblk, mkfs) and along the way things might not follow the exact steps from a guide. I asked dozens of questions about each tool and step, where previously I would have just "copy paste and prayed". Two nights ago I was similarly able to fully recover all my data from a failed ssd also using chatgpt to guide my learning along the way. It was really cool to tackle a completely new skill having a "guide" even if it's wrong 20% of the time, that's way better than the average on the open Internet. For someone who loves learning, it feels like thousand league boots compared to just endlessly sifting through internet crap. Of course everything it says is suspect, just like everything else on the Internet, but boy it cuts out a lot of the hassle.

iLoveOncall 2y ago
Okay but this is not programming, more about an enhanced tutorial. This isn't at all what the original commenter is talking about.
manmal 2y ago
You’ve given two examples for “broad”, but none for “deep”. I’ve also used LLMs for setting up my homelab, and they were really helpful since I was basically at beginner level in most linux admin topics (still am). But trying to eg setup automatic snapshot replications for my zfs pool had me go back to reading blog posts, as ChatGPT just couldn’t provide a solution that worked for me.
- Spivak 2y ago
  I think one of the catch-22s of LLMs is that using it as a fancy search index (which is the dev assistant use-case) is that the information it surfaces is hugely dependent on what words you use and it matches energy. If you don't know the words you'll get very beginner oriented content and you can get it to surface deeper knowledge iff you know the shibboleths, but it's annoying. One dumb trick that's been unreasonably useful is just copy-pasting barely-related source code, but at the level you're trying to understand and then just asking an unrelated question— just yank that search vector way over to the region where you think the good information lives.