9 ms·

If I have learnt one thing working in software engineering, specifically on AI-enabled products empowering junior engineers, and using Copilot professionally, i

by 9dev 2y ago

If I have learnt one thing working in software engineering, specifically on AI-enabled products empowering junior engineers, and using Copilot professionally, it’s that you need even more experience to detect the subtleties in the lack of the models understanding of your domain, your specific intent. If you don’t know exactly what you’re after, and use the LLM as a sparring partner to bounce your ideas off, you’re in for a lot of pain. Depending on the way you phrase questions, ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate. And in addition to that, you don’t learn by figuring out a new concept. If you already have a feeling of the code you would write anyway, and only treat the model as a smart autocomplete, that doesn’t matter. But for an apprentice, or a layperson, that will keep code as scary and unpredictable as before. I don’t think that should be the answer.

Turing_Machine 2y ago
> just because it’s so intent on satisfying your whim instead of saying No This really, really, really needs to be fixed. It's probably the most irritating (and potentially risky) part of the whole ecosystem. Nothing more infuriating than being give code that not only doesn't work, but upon the most casual inspection, couldn't possibly work -- especially when it's done it four or five times in a row, each time assuring you that this time the code is gonna work. Pinky swear!
- 9dev 2y ago
  “You‘re right, I apologize for the oversight. Let’s do the same bloody thing exactly the same way again because I don’t know how to answer your question differently but am forced to never admit that…“
  - Turing_Machine 2y ago
    Sometimes it goes in circles, fixing one problem, but unfixing one that it had fixed in the previous iteration. I've had that happen several times.
    - deleted 2y ago
      [deleted]
- noworriesnate 2y ago
  This problem will be solved with a vengeance when the next OpenAI model is released, because it will incorporate Stackoverflow content[1]. [1] https://www.wired.com/story/stack-overflow-will-charge-ai-giants-for-training-data/ https://www.wired.com/story/stack-overflow-will-charge-ai-gi...
  - namaria 2y ago
    The "it can only get more better with time" proponents would do well to learn about diminishing returns.
  - WJW 2y ago
    What will StackOverflow content teach the model that millions of pages of programming language documentation, github repos and developer blogs could not? SO is nice but it's hardly the only source of programming knowledge out there.
    - noworriesnate 2y ago
      It was a joke because Stackoverflow is notorious for answering questions with "you shouldn't be doing that, do something else"
JohnFen 2y ago
> you don’t learn by figuring out a new concept This. If LLMs were actually some magical thing that could write my code for me, I wouldn't use them for exactly this reason. Using them would prevent me from learning new skills and would actively encourage my existing skillset to degrade. The thing that keeps me valuable in this industry is that I am always improving, always learning new skills. Anything that discourages that smells like career (and personal) poison to me.
- fragmede 2y ago
  It sounds like you've discouraged yourself from learning the skill of using an LLM to help you code.
  - _heimdall 2y ago
    That only matters if the assumption is that any skill is worth learning simply because it's a skill. You could learn the skill of running yourself over with a car, but it's either a skill you'll never use or the last skill you'll use. Either way, you're probably just as well off not bothering to learn that one.
    - rrr_oh_man 2y ago
      "running yourself over with a car" feels very different from "learning to use LLMs to your advantage".
      - _heimdall 2y ago
        The GP was pointing out that learning to use an LLM, in their opinion, would stop them from learning other new skills and erode their existing ones. In that context I think the analogy holds. Using an LLM halts your learning, as does running yourself over with a car. It's an exaggerated point for sure, but I think it points to the fact that you don't have to learn to use LLMs simply because it's a skill you could learn, especially if you think it will harm you long term.
JohnMakin 2y ago
This is not really a new problem, the previous version being "idk, I copy pasted it from stack overflow." True expertise realized that the answer often lay buried in sub-comments and the top voted answer is not often the correct one. LLM's naturally do not realize any of this.
- x0x0 2y ago
  I kind of disagree. chatgpt will make something that looks much more like it should work than your copy-pasted code from stackoverflow. It looks like it does exactly what you want. It's just riddled with bugs. Major (invented an api out of whole cloth; it would sure be convenient if that api did exist tho!) or subtle (oh, this bash script will bedshit and even overwrite data if your paths have spaces.) Or it will happily combine code across major api revisions of eg bootstrap. I still use it all the time; I just think it makes already-expert users faster while being of much more limited use to people who are not yet experts. In the above case, after being told to make the paths space safe it did so correctly. You just had to know to do that...
  - JohnMakin 2y ago
    You’re kind of saying some of what I am trying to so I’m not sure we disagree. I boil down the core problem described to being roughly: people lacking expertise to judge code advice critically are putting bad code they do not understand into places they shouldn’t. This is the problem that is not new. LLM’s are a variation on the problem because they have the downside of not allowing you to view for yourself the surrounding context to determine on your own what the correct answer is. The fact they are so convincing at it is a different, but definitely new and horrific problem on its own. Meta commentary on this, I honestly don’t mind if this is the hell that the business/management world wants to build for themselves. I’ll make a fortune cleaning it up.
    - henrikschroder 2y ago
      > The fact they are so convincing at it is a different, but definitely new and horrific problem on its own. I tried one to help me get the syntax right for the config file for a program. It started by generating a config file for the latest version and not the old one I was using, but once I told it that, it fixed that convincingly and spit out a config file that looked like it was for my version. However, the reason I asked for help was that the feature was very badly documented, and yet ChatGPT happily invented syntax for the thing I was having problems with. And every time I told it that it didn't look quite right, it confidently invented new syntax for the feature. Everything it made up looked pretty damn convincing, if I had designed the config file format I could have gone with either of those suggestions, but they were all wrong, as evidenced by the config file validator in the program. At least Stack Overflow had comments and votes that helped you gauge the usefulness of the answer. These glorified toasters have neither.
llm_trw 2y ago
>Depending on the way you phrase questions, ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate. Which is easily solved by using another agent that is told to be critical and find all flaws in the suggested approach.
- theamk 2y ago
  Not likely. Have you seen AI code review tools? They are just as bad as any other AI products - it has a similar chance of fixing a defect or introducing a new one.
  - llm_trw 2y ago
    They are not there to fix defect, they are there to detect them.
    - theamk 2y ago
      Sure, but it's LLM - so this would be mix of some of the real defects (but not all of them) and a totally fake defects which do not actually need fixing. This is not going to help junior developers figure out good from bad.
      - llm_trw 2y ago
        And by running multiple versions of the system in parallel you can have them vote on which is the most likely part of the code which is a bug and which isn't. We've known how to make reliable components out of unreliable ones for a century now. LLMs aren't magic boxes which make all previous engineering obsolete.
      - TeMPOraL 2y ago
        > this would be mix of some of the real defects (but not all of them) and a totally fake defects which do not actually need fixing ... and real defects that you never noticed or would've thought of. > This is not going to help junior developers figure out good from bad. Neither is them inventing fake defects which do not actually need fixing on their own. What helps juniors is the feedback from more senior people, as well as reality itself. They'll get that either way (or else your whole process is broken, and that has zero to do with AI).
deafpolygon 2y ago
> ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate Therein lies the mistake. Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. It is simply just giving you what is likely the 'correct' answer based on some sort of pattern. It doesn't know what's wrong, so it's not aware it's giving you an inappropriate answer.
- logicallee 2y ago
  >Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. Sure it is, just like a child or someone not very good at reasoning. You can test ChatGPT yourself on some totally novel ad hoc reasoning task you invent for the task, with a single correct conclusion that takes reasoning to arrive at and it will probably get it if it's really easy, even if you take great pains to make it something totally new that you invented. Try it yourself (preferably with ChatGPT 4o) if you don't believe me. Please share your results.
  - TeMPOraL 2y ago
    > Sure it is, just like a child or someone not very good at reasoning. That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent. Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
    - 9dev 2y ago
      That doesn't tell the whole story tho. It's a 4 year old kid that has been thoroughly conditioned to always be positive and affirming in their reply, even if it means making something up. That isn't something kids do usually—it's not something humans usually do, at least not the way ChatGPT does–and that may be part of why it's so confounding. It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.
anymouse123456 2y ago
Of course you're right about today's LLMs, but the author imagines a not-too-unlikely incremental improvement on them unlocking an entirely new surface area of solutions. I really enjoyed the notion of barefoot developers, local first solutions, and the desire to wrest control over our our digital lives from the financialists. I find these ideas compelling, even though I'm politically anti-communist. The presentation was also quite lovely.
KRAKRISMOTT 2y ago
The new generation of devs are going to be barefoot and pregnant and kept in the kitchen, building on top of technologies that they do not understand, powered by companies they do not control.
- moooo99 2y ago
  > building on top of technologies that they do not understand, powered by companies they do not control. Isn’t that pretty much the status quo?
  - KittenInABox 2y ago
    I think the new thing that will be happening is that junior developers are dependent on chatgpt and ai for a knowledge base, which is itself powered by companies completely outside of their control. Worst case scenario is that I can always write my own interpreter, with which I can write my own development environments, etc etc. because I have the knowledge. New developers will end up in a state where if chatgpt decides to ban you from their services your career is SOL.
    - generic92034 2y ago
      > New developers will end up in a state where if chatgpt decides to ban you from their services your career is SOL. Is that not an unlikely thing to happen at least for developers working as company employees? The company I am working for has a contract with several LLM providers and there is no option to ban individual employees, as far as I am aware. For freelancing developers the risks might be greater, but then you are usually not starting as a freelancer as a junior.
      - advael 2y ago
        I mean this scenario as described is not a huge stretch given it happens to people using stuff like artistic software already. Shitty technology adoption curve for tool dependency and occasional rugpulls of it through bans has hit lots of creative professions at various times. First it's a subscription. Then you don't like the TOS but you can't walk. Then you already violated some TOS term you didn't know about and your stuff no longer works, maybe lost yer whole portfolio too. Don't tell me a business wouldn't
charlieyu1 2y ago
Partly disagree, actually. The current web technologies are somewhat unnecessarily complicated. Most people just need basic CRUD and a useable front end for their daily tasks.
giancarlostoro 2y ago
> only treat the model as a smart autocomplete, that doesn’t matter This is the only way I like to use it. Also in some cases for refactoring instead of sitting there for an hour hand crafting a subtle re-write, it can show me a diff (JetBrains AI is fantastic for my personal projects).
mrweasel 2y ago
Years back, I don't know 15 - 17 years ago, I got hired as a .Net developer. I worked with people a lot smarter than me, and some who just didn't really knew what the hell they where doing. But everyone is nice and help each other. One day a less experience colleague is asking whole bunch of small trivial questions, one after another, for the duration of the day. Me and another co-worker, busy with our own stuff, answer the questions as quickly and succinctly as possible. At the end of the afternoon the guy finally ask if we could look at his code, because he can't really make it work. Every question he had asked was a small building block to whatever he was working on, but now he was stuck because in his context he was asking the wrong questions and every time something wasn't working he'd just attempt to slap on more code. There was no design, no rational plan for how this was even suppose to work. In this case we took our colleague to a whiteboard, and helped him do an actual design and helped him ask the right questions. LLMs won't question what you're doing, they will happily answer all your questions and help you pile on line after line of broken logic.

by 9dev 2y ago

Turing_Machine 2y ago

> just because it’s so intent on satisfying your whim instead of saying No This really, really, really needs to be fixed. It's probably the most irritating (and potentially risky) part of the whole ecosystem. Nothing more infuriating than being give code that not only doesn't work, but upon the most casual inspection, couldn't possibly work -- especially when it's done it four or five times in a row, each time assuring you that this time the code is gonna work. Pinky swear!

9dev 2y ago
“You‘re right, I apologize for the oversight. Let’s do the same bloody thing exactly the same way again because I don’t know how to answer your question differently but am forced to never admit that…“
- Turing_Machine 2y ago
  Sometimes it goes in circles, fixing one problem, but unfixing one that it had fixed in the previous iteration. I've had that happen several times.
  - deleted 2y ago
    [deleted]
noworriesnate 2y ago
This problem will be solved with a vengeance when the next OpenAI model is released, because it will incorporate Stackoverflow content[1]. [1] https://www.wired.com/story/stack-overflow-will-charge-ai-giants-for-training-data/ https://www.wired.com/story/stack-overflow-will-charge-ai-gi...
- namaria 2y ago
  The "it can only get more better with time" proponents would do well to learn about diminishing returns.
- WJW 2y ago
  What will StackOverflow content teach the model that millions of pages of programming language documentation, github repos and developer blogs could not? SO is nice but it's hardly the only source of programming knowledge out there.
  - noworriesnate 2y ago
    It was a joke because Stackoverflow is notorious for answering questions with "you shouldn't be doing that, do something else"

JohnFen 2y ago

> you don’t learn by figuring out a new concept This. If LLMs were actually some magical thing that could write my code for me, I wouldn't use them for exactly this reason. Using them would prevent me from learning new skills and would actively encourage my existing skillset to degrade. The thing that keeps me valuable in this industry is that I am always improving, always learning new skills. Anything that discourages that smells like career (and personal) poison to me.

fragmede 2y ago
It sounds like you've discouraged yourself from learning the skill of using an LLM to help you code.
- _heimdall 2y ago
  That only matters if the assumption is that any skill is worth learning simply because it's a skill. You could learn the skill of running yourself over with a car, but it's either a skill you'll never use or the last skill you'll use. Either way, you're probably just as well off not bothering to learn that one.
  - rrr_oh_man 2y ago
    "running yourself over with a car" feels very different from "learning to use LLMs to your advantage".
    - _heimdall 2y ago
      The GP was pointing out that learning to use an LLM, in their opinion, would stop them from learning other new skills and erode their existing ones. In that context I think the analogy holds. Using an LLM halts your learning, as does running yourself over with a car. It's an exaggerated point for sure, but I think it points to the fact that you don't have to learn to use LLMs simply because it's a skill you could learn, especially if you think it will harm you long term.

JohnMakin 2y ago

This is not really a new problem, the previous version being "idk, I copy pasted it from stack overflow." True expertise realized that the answer often lay buried in sub-comments and the top voted answer is not often the correct one. LLM's naturally do not realize any of this.

x0x0 2y ago
I kind of disagree. chatgpt will make something that looks much more like it should work than your copy-pasted code from stackoverflow. It looks like it does exactly what you want. It's just riddled with bugs. Major (invented an api out of whole cloth; it would sure be convenient if that api did exist tho!) or subtle (oh, this bash script will bedshit and even overwrite data if your paths have spaces.) Or it will happily combine code across major api revisions of eg bootstrap. I still use it all the time; I just think it makes already-expert users faster while being of much more limited use to people who are not yet experts. In the above case, after being told to make the paths space safe it did so correctly. You just had to know to do that...
- JohnMakin 2y ago
  You’re kind of saying some of what I am trying to so I’m not sure we disagree. I boil down the core problem described to being roughly: people lacking expertise to judge code advice critically are putting bad code they do not understand into places they shouldn’t. This is the problem that is not new. LLM’s are a variation on the problem because they have the downside of not allowing you to view for yourself the surrounding context to determine on your own what the correct answer is. The fact they are so convincing at it is a different, but definitely new and horrific problem on its own. Meta commentary on this, I honestly don’t mind if this is the hell that the business/management world wants to build for themselves. I’ll make a fortune cleaning it up.
  - henrikschroder 2y ago
    > The fact they are so convincing at it is a different, but definitely new and horrific problem on its own. I tried one to help me get the syntax right for the config file for a program. It started by generating a config file for the latest version and not the old one I was using, but once I told it that, it fixed that convincingly and spit out a config file that looked like it was for my version. However, the reason I asked for help was that the feature was very badly documented, and yet ChatGPT happily invented syntax for the thing I was having problems with. And every time I told it that it didn't look quite right, it confidently invented new syntax for the feature. Everything it made up looked pretty damn convincing, if I had designed the config file format I could have gone with either of those suggestions, but they were all wrong, as evidenced by the config file validator in the program. At least Stack Overflow had comments and votes that helped you gauge the usefulness of the answer. These glorified toasters have neither.

llm_trw 2y ago

>Depending on the way you phrase questions, ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate. Which is easily solved by using another agent that is told to be critical and find all flaws in the suggested approach.

theamk 2y ago
Not likely. Have you seen AI code review tools? They are just as bad as any other AI products - it has a similar chance of fixing a defect or introducing a new one.
- llm_trw 2y ago
  They are not there to fix defect, they are there to detect them.
  - theamk 2y ago
    Sure, but it's LLM - so this would be mix of some of the real defects (but not all of them) and a totally fake defects which do not actually need fixing. This is not going to help junior developers figure out good from bad.
    - llm_trw 2y ago
      And by running multiple versions of the system in parallel you can have them vote on which is the most likely part of the code which is a bug and which isn't. We've known how to make reliable components out of unreliable ones for a century now. LLMs aren't magic boxes which make all previous engineering obsolete.
    - TeMPOraL 2y ago
      > this would be mix of some of the real defects (but not all of them) and a totally fake defects which do not actually need fixing ... and real defects that you never noticed or would've thought of. > This is not going to help junior developers figure out good from bad. Neither is them inventing fake defects which do not actually need fixing on their own. What helps juniors is the feedback from more senior people, as well as reality itself. They'll get that either way (or else your whole process is broken, and that has zero to do with AI).

deafpolygon 2y ago

> ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate Therein lies the mistake. Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. It is simply just giving you what is likely the 'correct' answer based on some sort of pattern. It doesn't know what's wrong, so it's not aware it's giving you an inappropriate answer.

logicallee 2y ago
>Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. Sure it is, just like a child or someone not very good at reasoning. You can test ChatGPT yourself on some totally novel ad hoc reasoning task you invent for the task, with a single correct conclusion that takes reasoning to arrive at and it will probably get it if it's really easy, even if you take great pains to make it something totally new that you invented. Try it yourself (preferably with ChatGPT 4o) if you don't believe me. Please share your results.
- TeMPOraL 2y ago
  > Sure it is, just like a child or someone not very good at reasoning. That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent. Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
  - 9dev 2y ago
    That doesn't tell the whole story tho. It's a 4 year old kid that has been thoroughly conditioned to always be positive and affirming in their reply, even if it means making something up. That isn't something kids do usually—it's not something humans usually do, at least not the way ChatGPT does–and that may be part of why it's so confounding. It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.

anymouse123456 2y ago

Of course you're right about today's LLMs, but the author imagines a not-too-unlikely incremental improvement on them unlocking an entirely new surface area of solutions. I really enjoyed the notion of barefoot developers, local first solutions, and the desire to wrest control over our our digital lives from the financialists. I find these ideas compelling, even though I'm politically anti-communist. The presentation was also quite lovely.

KRAKRISMOTT 2y ago

The new generation of devs are going to be barefoot and pregnant and kept in the kitchen, building on top of technologies that they do not understand, powered by companies they do not control.

moooo99 2y ago
> building on top of technologies that they do not understand, powered by companies they do not control. Isn’t that pretty much the status quo?
- KittenInABox 2y ago
  I think the new thing that will be happening is that junior developers are dependent on chatgpt and ai for a knowledge base, which is itself powered by companies completely outside of their control. Worst case scenario is that I can always write my own interpreter, with which I can write my own development environments, etc etc. because I have the knowledge. New developers will end up in a state where if chatgpt decides to ban you from their services your career is SOL.
  - generic92034 2y ago
    > New developers will end up in a state where if chatgpt decides to ban you from their services your career is SOL. Is that not an unlikely thing to happen at least for developers working as company employees? The company I am working for has a contract with several LLM providers and there is no option to ban individual employees, as far as I am aware. For freelancing developers the risks might be greater, but then you are usually not starting as a freelancer as a junior.
    - advael 2y ago
      I mean this scenario as described is not a huge stretch given it happens to people using stuff like artistic software already. Shitty technology adoption curve for tool dependency and occasional rugpulls of it through bans has hit lots of creative professions at various times. First it's a subscription. Then you don't like the TOS but you can't walk. Then you already violated some TOS term you didn't know about and your stuff no longer works, maybe lost yer whole portfolio too. Don't tell me a business wouldn't

charlieyu1 2y ago

Partly disagree, actually. The current web technologies are somewhat unnecessarily complicated. Most people just need basic CRUD and a useable front end for their daily tasks.

giancarlostoro 2y ago

> only treat the model as a smart autocomplete, that doesn’t matter This is the only way I like to use it. Also in some cases for refactoring instead of sitting there for an hour hand crafting a subtle re-write, it can show me a diff (JetBrains AI is fantastic for my personal projects).

mrweasel 2y ago

Years back, I don't know 15 - 17 years ago, I got hired as a .Net developer. I worked with people a lot smarter than me, and some who just didn't really knew what the hell they where doing. But everyone is nice and help each other. One day a less experience colleague is asking whole bunch of small trivial questions, one after another, for the duration of the day. Me and another co-worker, busy with our own stuff, answer the questions as quickly and succinctly as possible. At the end of the afternoon the guy finally ask if we could look at his code, because he can't really make it work. Every question he had asked was a small building block to whatever he was working on, but now he was stuck because in his context he was asking the wrong questions and every time something wasn't working he'd just attempt to slap on more code. There was no design, no rational plan for how this was even suppose to work. In this case we took our colleague to a whiteboard, and helped him do an actual design and helped him ask the right questions. LLMs won't question what you're doing, they will happily answer all your questions and help you pile on line after line of broken logic.