4 ms·

People who use ChatGPT to spit out code as a method of learning software engineering are going to be seriously deficient in fundamentals like performance. I've

People who use ChatGPT to spit out code as a method of learning software engineering are going to be seriously deficient in fundamentals like performance. I've already worked with way too many engineers who copy/paste code without understanding what it does.

stasmo 3y ago
The good news is that those copy/pasters are probably going to be copy/pasting higher quality code if they’re using GPT4.
- hoorayimhelping 3y ago
  I don't think this is good news, it's at best not-bad news. We're already inundated with content, most of it copycatting other content, and it's replacing meaningful things, like archetypal stories and philosophy. Look at how people describe Marvel movies and Star Wars shows - it's content, not stories. Scaling the ability to just repeat things without an understanding of those things will at best just add more content. It will likely contribute to the further creation of complex systems people don't understand and can't control. "Um, I'll tell you the problem with the scientific power that you're, that you're using here. It didn't require any discipline to attain it. Ya know, you read what others had done, and you, and you took took the next step. You didn't earn the knowledge for yourselves, so you don't take any responsibility for it. You stood on the shoulders of geniuses, uh, to accomplish something as fast as you could..." (Emphasis mine). https://www.youtube.com/watch?v=4PLvdmifDSk&t=61s https://www.youtube.com/watch?v=4PLvdmifDSk&t=61s
  - fleetwoodsnack 3y ago
    Just about any original scientific text or undertaking begins with a literature review.
nivenkos 3y ago
But the awesome part is that you can ask it to explain parts and it usually does pretty well. I was using it for debugging WINE - where I have no DirectDraw4 / DirectX6 experience, and it's much nicer than trying to trawl through ancient MS documentation myself.
simonw 3y ago
I think this is currently an open question. It may turn out that ChatGPT results in a new generation of developers who produce terrible code, full of flaws, with no understanding of how it actually works. Or... it might be that ChatGPT helps newcomers get over that horrendous initial learning curve, start by writing bad code... and then get better at it. I'm currently cautiously optimistic that the latter scenario is going to win out.
- patapong 3y ago
  I share your optimism. ChatGPT could also empower individuals who may not have considered learning programming to create practical programs that enhance their daily routines, similar to how VB6 significantly reduced the barrier to entry for developing various applications.
- protosevn 3y ago
  I'm on year two of my journey to become a game developer, i am focusing on programming and if i realize i don´t want to work with games in the future at least i have solid knowledge on how to deal with a broad area of software problems. I don´t rely on ChatGPT to write any code for me but i do use it whenever there´s a concept i don´t understand and if there´s any problem blocking me from getting forward in my many programming books. Having a tireless teacher/mentor with extensive knowledge and who does not care however stupid question you ask is a very good tool for learning.
- sha016 3y ago
  I don't think it will help people learn as much as having to write everything yourself. Sure it will get them productive 10x faster, as a shortcut. You can only get so good at riding a bicycle by watching others ride a bike. Remember that caching issue that cost you hours of debugging? You don't forget that. Maybe the generated code doesn't produce a bug this time, but the 5th or 6th time? It's hard enough to debug code you wrote yourself sometimes, let alone generated code.
- ska 3y ago
  In my experience newcomers and junior engineers don't get better by writing more bad code. They learn mostly by 3 methods. 1) finding flaws in their code the hard way (slow) 2) getting careful feedback from more experienced people (has to be done well) and 3) reading better code for a problem they already understand. I'm not sure GPT helps with any of those yet, as it seems to be more likely to spit out incorrect-to-sortof-okayish so far.
giobox 3y ago
I've seen similar results with teams I work with. The strongest engineers who can spot the deficiencies or errors in LLM generated code generally get a huge productivity boost, as they can compare the "shape" of the output to what they would expect for a given problem. The junior devs, lacking the experience that provides the context or knowledge to analyze the "shape", let all manner of minor and major issues from the generated code slip past. I really notice this particularly with say less popular frameworks or code libraries, where many LLM code generators appear much more likely to hallucinate invalid code. I see similar issues with DSLs too, where the LLM starts to just guess at the DSL syntax. I really hate when an engineer new to a framework just goes to the LLM first, without at least spending a little time learning the framework for the task. All said, I suspect these code generation tools will get good enough that the rate of errors goes down enough it maybe won't matter in the longer term.
stevenhuang 3y ago
If performance is a concern, just re-prompt GPT if there are any performance concerns, and to propose fixes for them if there are. This is called reflexion and has been found to be an effective way to improve response accuracy and usefulness (also reduces hallucinations). https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366

stasmo 3y ago

The good news is that those copy/pasters are probably going to be copy/pasting higher quality code if they’re using GPT4.

hoorayimhelping 3y ago
I don't think this is good news, it's at best not-bad news. We're already inundated with content, most of it copycatting other content, and it's replacing meaningful things, like archetypal stories and philosophy. Look at how people describe Marvel movies and Star Wars shows - it's content, not stories. Scaling the ability to just repeat things without an understanding of those things will at best just add more content. It will likely contribute to the further creation of complex systems people don't understand and can't control. "Um, I'll tell you the problem with the scientific power that you're, that you're using here. It didn't require any discipline to attain it. Ya know, you read what others had done, and you, and you took took the next step. You didn't earn the knowledge for yourselves, so you don't take any responsibility for it. You stood on the shoulders of geniuses, uh, to accomplish something as fast as you could..." (Emphasis mine). https://www.youtube.com/watch?v=4PLvdmifDSk&t=61s https://www.youtube.com/watch?v=4PLvdmifDSk&t=61s
- fleetwoodsnack 3y ago
  Just about any original scientific text or undertaking begins with a literature review.

nivenkos 3y ago

But the awesome part is that you can ask it to explain parts and it usually does pretty well. I was using it for debugging WINE - where I have no DirectDraw4 / DirectX6 experience, and it's much nicer than trying to trawl through ancient MS documentation myself.

simonw 3y ago

I think this is currently an open question. It may turn out that ChatGPT results in a new generation of developers who produce terrible code, full of flaws, with no understanding of how it actually works. Or... it might be that ChatGPT helps newcomers get over that horrendous initial learning curve, start by writing bad code... and then get better at it. I'm currently cautiously optimistic that the latter scenario is going to win out.

patapong 3y ago
I share your optimism. ChatGPT could also empower individuals who may not have considered learning programming to create practical programs that enhance their daily routines, similar to how VB6 significantly reduced the barrier to entry for developing various applications.
protosevn 3y ago
I'm on year two of my journey to become a game developer, i am focusing on programming and if i realize i don´t want to work with games in the future at least i have solid knowledge on how to deal with a broad area of software problems. I don´t rely on ChatGPT to write any code for me but i do use it whenever there´s a concept i don´t understand and if there´s any problem blocking me from getting forward in my many programming books. Having a tireless teacher/mentor with extensive knowledge and who does not care however stupid question you ask is a very good tool for learning.
sha016 3y ago
I don't think it will help people learn as much as having to write everything yourself. Sure it will get them productive 10x faster, as a shortcut. You can only get so good at riding a bicycle by watching others ride a bike. Remember that caching issue that cost you hours of debugging? You don't forget that. Maybe the generated code doesn't produce a bug this time, but the 5th or 6th time? It's hard enough to debug code you wrote yourself sometimes, let alone generated code.
ska 3y ago
In my experience newcomers and junior engineers don't get better by writing more bad code. They learn mostly by 3 methods. 1) finding flaws in their code the hard way (slow) 2) getting careful feedback from more experienced people (has to be done well) and 3) reading better code for a problem they already understand. I'm not sure GPT helps with any of those yet, as it seems to be more likely to spit out incorrect-to-sortof-okayish so far.

giobox 3y ago

I've seen similar results with teams I work with. The strongest engineers who can spot the deficiencies or errors in LLM generated code generally get a huge productivity boost, as they can compare the "shape" of the output to what they would expect for a given problem. The junior devs, lacking the experience that provides the context or knowledge to analyze the "shape", let all manner of minor and major issues from the generated code slip past. I really notice this particularly with say less popular frameworks or code libraries, where many LLM code generators appear much more likely to hallucinate invalid code. I see similar issues with DSLs too, where the LLM starts to just guess at the DSL syntax. I really hate when an engineer new to a framework just goes to the LLM first, without at least spending a little time learning the framework for the task. All said, I suspect these code generation tools will get good enough that the rate of errors goes down enough it maybe won't matter in the longer term.

stevenhuang 3y ago

If performance is a concern, just re-prompt GPT if there are any performance concerns, and to propose fixes for them if there are. This is called reflexion and has been found to be an effective way to improve response accuracy and usefulness (also reduces hallucinations). https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366