6 ms·

> Now I just assume they're taking my feedback and feeding it right back to the LLM. This is especially annoying when you get back a response in a PR "Yes, you

by shagie 11mo ago

> Now I just assume they're taking my feedback and feeding it right back to the LLM. This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested." Part of the challenge (and I don't have an answer either) is there are some juniors who use AI to assist... and some who use it to delegate all of their work to. It is especially frustrating that the second group doesn't become much more than a proxy for an LLM. New juniors can progress in software engineering - but they have to take the road of disciplined use of AI and make sure that they're learning the material rather than delegating all their work to it... and that delegating work is very tempting... especially if that's what they did in college.

mooreds 11mo ago
> there are some juniors who use AI to assist... and some who use it to delegate all of their work to. Hmmm. Is there any way to distinguish between these two categories? Because I agree, if someone is delegating all their work to an LLM or similar tool, cut out the middleman. Same as if someone just copy/pasted from Stackoverflow 5 years ago. I think it is also important to think about incentives. What incentive does the newer developer have to understand the LLM output? There's the long term incentive, but is there a short term one?
- supriyo-biswas 11mo ago
  Dealing with an intern at work who I suspect is doing exactly this, I discussed this with a colleague. One way seems to be to organize a face to face meeting where you test their problem solving skills without AI use, the other may be to question them about their thought process as you review a PR. Unfortunately, the use of LLMs has brought about a lot of mistrust in the workplace. Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice as they see it as sermonizing when an “easy” process to get “acceptable” results exists.
  - throwaway2037 11mo ago
    > Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice Hot take: This reads like an old person looking down upon young people. Can you explain why it isn't? Else, this reads like: "When I was young, we worked hard and listened to our elders. These days, young people ignore our advice." Every time I see inter-generational commentary like this (which is inevitably from personal experience), I am immediately suspicious. I can assure you that when I was young, I did not listen to older people's advice and I tried to do everything my own way. Why would this be any different in the current generation? In my experience, it isn't. On a positive note: I can remember mentoring some young people and watching them comb through blogs to learn about programming. I am so old that my shelf is/was full of O'Reilly books. By the time I was mentoring them, few people under 25 were reading O'Reilly books. It opened my eyes that how people changes more than what people learn. Example: Someone is trying to learning about access control modifiers for classes/methods in a programming language. Old days: Get the O'Reilly book for that programming language. Lookup access modifiers in the index. 10 year ago: Google for a blog with an intro to the programming language. There will be a tip about what access modifiers can do. Today: Ask ChatGPT. In my (somewhat contrived) example, the how is changing, but not the what.
    - transfer92 11mo ago
      > I can assure you that when I was young, I did not listen to older people's advice and I tried to do everything my own way. Hot take: This reads like a person who was difficult to work with. Senior people have responsibility, therefore in a business situation they have authority. Junior people who think they know it all don't like this. If there's a disagreement between a senior person and a junior person about something, they should, of course, listen to each other respectfully. If that's not happening, then one of them is not being a good employee. But if they are, then the supervisor makes the final call.
    - shagie 11mo ago
      > Old days: Get the O'Reilly book for that programming language. Lookup access modifiers in the index. 10 year ago: Google for a blog with an intro to the programming language. There will be a tip about what access modifiers can do. Today: Ask ChatGPT. In my (somewhat contrived) example, the how is changing, but not the what. The tangent to that is it is also changing with the how much one internalizes about the problem domain and is able to apply that knowledge later. Hard fought knowledge from the old days is something that shapes how I design systems today. However, the tendency of people who reach for ChatGPT today to solve a problem results in them making the same mistakes again the next time since the information is so easy to access. It also results in things that are larger are more difficult... the "how do you architect this larger system" is something you learn by building the smaller systems and learning about them so that their advantages and disadvantages and how and such becomes an inherent part of how you conceive of the system as a whole. ... Being able to have ChatGPT do it means people often don't think about the larger problem or how it fits together. I believe that is harder for a junior who is using ChatGPT to advance to being a mid level or senior developer than it is for a junior from the old days because of the lack of retention of the knowledge of the problems and solutions.
- lll-o-lll 11mo ago
  > Is there any way to distinguish between these two categories? Yes, it should be obvious. At least at the current state of LLMs. > There's the long term incentive, but is there a short term one? The short term incentive is keeping their job.
- hombre_fatal 11mo ago
  Just like anything, anyone who did the work themself should be able to speak intelligently about the work and the decisions behind its idiosyncrasies. For software, I can imagine a process where junior developers create a PR and then run through it with another engineer side by side. The short-term incentive would be that they can do it, else they'd get exposed.
- icedchai 11mo ago
  There are some definite signs of over reliance on AI. From emojis in comments, to updates completely unrelated to the task at hand, if you ask "why did you make this change?", you'll typically get no answer. I don't mind if AI is used as a tool, but the output needs to be vetted.
  - wwweston 11mo ago
    Exactly. Use LLMs as a tutor, a tool, and make sure you understand the output.
    - agumonkey 11mo ago
      My favorite prompt is "your goal is to retire yourself"
  - throwaway2037 11mo ago
    What is wrong with emojis in comments? I see no issue with it. Do I do it myself? No. Would I pushback if a young person added emojis to comments? No. I am looking at "the content, not the colour".
    - chihuahua 11mo ago
      I think GP may be thinking that emojis in PR comments (plus the other red flags they mentioned) are the result of copy/paste from LLM output, which might imply that the person who does mindless copy/pasting is not adding anything and could be replaced by LLM automation.
    - venturecruelty 11mo ago
      The point is that heavy emoji use means AI was likely used to produce a changeset, not that emojis are inherently bad.
    - icedchai 11mo ago
      The emojis are not a problem themselves. They're a warning sign: slop is (probably) present, look deeper.
- bryanrasmussen 11mo ago
  having dealt with a few people who just copy/pasted Stackoverflow I really feel that using an LLM is an improvement. That is at least for the people who don't understand what they're doing, the LLM tends to come out with something I can at least turn into something useful. It might be reversed though for people who know what they're doing. IF they know what they're doing they might theoretically be able to put together some stackoverflow results that make sense, and build something up from that better than what gets generated from LLM (I am not asserting this would happen, and thinking it might be the case) However I don't know as I've never known anyone who knew what they were doing who also just copy/pasted some stackoverflow or delegated to LLM significantly.
- water-data-dude 11mo ago
  Is/was copy/pasting from Stackoverflow considered harmful? You have a problem, you do a web search and you find someone who asked the same question on SO, and there's often a solution. You might be specifically talking about people who copy/paste without understanding, but I think it's still OK-ish to do that, since you can't make an entire [whatever you're coding up] by copy/pasting snippets from SO like you're cutting words out of a magazine for a ransom note. There's still thought involved, so it's more like training wheels that you eventually outgrow as you get more understanding.
  - vkou 11mo ago
    > Is/was copy/pasting from Stackoverflow considered harmful? It at least forces you to tinker with whatever you copied over.
- gunsch 11mo ago
  Pair programming! Get hands-on with your junior engineers and their development process. Push them to think through things and not just ask the LLM everything.
  - johnnyanmac 11mo ago
    I've seen some overly excessive pair programming initiatives out there, but it does baffle me why less people who struggle with this do it. Take even just 30 minutes to pair program on a problem and see their process and you can reveal so much. But I suppose my question is rhetorical. We're laying off hundreds of thousands of engineers and maming existing ones do the work of 3-4 engineers. Not much time to help the juniors.
anal_reactor 11mo ago
> This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested." I've learnt that saying this exact phrase does wonders when it comes to advancing your career. I used to argue against stupid ideas but not only did I achieve nothing, but I was also labelled uncooperative and technically incompetent. Then I became a "yes-man" and all problems went away.
- throwaway2037 11mo ago
  This. May you have great success! My PR comments that I get are so dumb. I can put the most obvious bugs in my code, but people are focused in the colour of the bike shed. I am happy to repaint the bike shed whatever colour they need it to be!
- shagie 11mo ago
  I was attempting to mock Claude's "You are absolutely right" style of response when corrected. I have seen responses to PRs that appear to be a copy and paste of my feedback into it and a copy and paste of the response and fixes into the PR. It may be the that the developer is incorporating the mannerisms of Claude into their own speech... that would be something to delve into (that was intentional). However, more often than not in today's world of software development such responses are more likely to indicate a copy and paste of LLM generated content.
  - anal_reactor 11mo ago
    > However, more often than not in today's world of software development such responses are more likely to indicate a copy and paste of LLM generated content. This is nothing new. People rarely have independent thoughts, usually they just parrot whatever they've been told to parrot. LLMs created common world-wide standard on this parroting, which makes the phenomenon more evident, but it doesn't change the fact that it existed before LLMs. Have you ever had a conversation with an intelligent person and thought "wow that's refreshing"? Yeah. There's a reason why it feels so good.
johnnyanmac 11mo ago
I must ask once again why we are having these 5+ round interview cycles and we aren't able to filter for qualities that the work requires of its talent. What are all those rounds for if we're getting engineers who aren't as valued for the team's needs at the end of the pipeline?
- venturecruelty 11mo ago
  It's the cargo cult kayfabe of it all. People do it because Google used to do it, now it's just spread like a folk religion. But nobody wants guilds or licensure, so we have to make everyone do a week-long take-home and then FizzBuzz in front of a very awkward committee. Might as well just read chicken bones, at least that would be less humiliating.
  - johnnyanmac 11mo ago
    Yeah, I'd be more than fine with licensing if I didn't have to keep going through 5 rounds of trivia only to be ghosted. Let me do that once and show I can code my way out of a paper bag.
  - nradov 11mo ago
    And who would write the guild membership or licensure criteria? How much should those focus on ReactJS versus validation criteria for cruise missile flight control software?
    - throwup238 11mo ago
      Guild members? Who else? You’re asking these rhetorical questions as if we haven’t had centuries of precedent here, both bad and good. How does the AMA balance between neurosurgeons and optometrists? Bar associations between corporate litigators and family estate lawyers? Professional engineering associations between civil engineers and chemical engineers?
      - nradov 11mo ago
locknitpicker 11mo ago
> Part of the challenge (and I don't have an answer either) is there are some juniors who use AI to assist... and some who use it to delegate all of their work to. This is not limited to junior devs. I had the displeasure of working with a guy who was hired as a senior dev who heavily delegated any work they did. He failed to even do the faintest review of what the coding agent and of course did zero testing. At one time these stunts resulted in a major incident where one of these glorious PRs pushed code that completely inverted a key business rule and resulted in paying customers being denied access to a paid product. Sometimes people are slackers with little to no ownership or pride in their craftsmanship, and just stumbled upon a career path they are not very good at. They start at juniors but they can idle long enough to waddle their way to senior positions. This is not a LLM problem, or caused by it.
sevenseacat 11mo ago
> This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested." And then in the next PR, you have to request the exact same changes

by shagie 11mo ago

mooreds 11mo ago

> there are some juniors who use AI to assist... and some who use it to delegate all of their work to. Hmmm. Is there any way to distinguish between these two categories? Because I agree, if someone is delegating all their work to an LLM or similar tool, cut out the middleman. Same as if someone just copy/pasted from Stackoverflow 5 years ago. I think it is also important to think about incentives. What incentive does the newer developer have to understand the LLM output? There's the long term incentive, but is there a short term one?

supriyo-biswas 11mo ago
Dealing with an intern at work who I suspect is doing exactly this, I discussed this with a colleague. One way seems to be to organize a face to face meeting where you test their problem solving skills without AI use, the other may be to question them about their thought process as you review a PR. Unfortunately, the use of LLMs has brought about a lot of mistrust in the workplace. Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice as they see it as sermonizing when an “easy” process to get “acceptable” results exists.
- throwaway2037 11mo ago
  > Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice Hot take: This reads like an old person looking down upon young people. Can you explain why it isn't? Else, this reads like: "When I was young, we worked hard and listened to our elders. These days, young people ignore our advice." Every time I see inter-generational commentary like this (which is inevitably from personal experience), I am immediately suspicious. I can assure you that when I was young, I did not listen to older people's advice and I tried to do everything my own way. Why would this be any different in the current generation? In my experience, it isn't. On a positive note: I can remember mentoring some young people and watching them comb through blogs to learn about programming. I am so old that my shelf is/was full of O'Reilly books. By the time I was mentoring them, few people under 25 were reading O'Reilly books. It opened my eyes that how people changes more than what people learn. Example: Someone is trying to learning about access control modifiers for classes/methods in a programming language. Old days: Get the O'Reilly book for that programming language. Lookup access modifiers in the index. 10 year ago: Google for a blog with an intro to the programming language. There will be a tip about what access modifiers can do. Today: Ask ChatGPT. In my (somewhat contrived) example, the how is changing, but not the what.
  - transfer92 11mo ago
    > I can assure you that when I was young, I did not listen to older people's advice and I tried to do everything my own way. Hot take: This reads like a person who was difficult to work with. Senior people have responsibility, therefore in a business situation they have authority. Junior people who think they know it all don't like this. If there's a disagreement between a senior person and a junior person about something, they should, of course, listen to each other respectfully. If that's not happening, then one of them is not being a good employee. But if they are, then the supervisor makes the final call.
  - shagie 11mo ago
    > Old days: Get the O'Reilly book for that programming language. Lookup access modifiers in the index. 10 year ago: Google for a blog with an intro to the programming language. There will be a tip about what access modifiers can do. Today: Ask ChatGPT. In my (somewhat contrived) example, the how is changing, but not the what. The tangent to that is it is also changing with the how much one internalizes about the problem domain and is able to apply that knowledge later. Hard fought knowledge from the old days is something that shapes how I design systems today. However, the tendency of people who reach for ChatGPT today to solve a problem results in them making the same mistakes again the next time since the information is so easy to access. It also results in things that are larger are more difficult... the "how do you architect this larger system" is something you learn by building the smaller systems and learning about them so that their advantages and disadvantages and how and such becomes an inherent part of how you conceive of the system as a whole. ... Being able to have ChatGPT do it means people often don't think about the larger problem or how it fits together. I believe that is harder for a junior who is using ChatGPT to advance to being a mid level or senior developer than it is for a junior from the old days because of the lack of retention of the knowledge of the problems and solutions.
lll-o-lll 11mo ago
> Is there any way to distinguish between these two categories? Yes, it should be obvious. At least at the current state of LLMs. > There's the long term incentive, but is there a short term one? The short term incentive is keeping their job.
hombre_fatal 11mo ago
Just like anything, anyone who did the work themself should be able to speak intelligently about the work and the decisions behind its idiosyncrasies. For software, I can imagine a process where junior developers create a PR and then run through it with another engineer side by side. The short-term incentive would be that they can do it, else they'd get exposed.
icedchai 11mo ago
There are some definite signs of over reliance on AI. From emojis in comments, to updates completely unrelated to the task at hand, if you ask "why did you make this change?", you'll typically get no answer. I don't mind if AI is used as a tool, but the output needs to be vetted.
- wwweston 11mo ago
  Exactly. Use LLMs as a tutor, a tool, and make sure you understand the output.
  - agumonkey 11mo ago
    My favorite prompt is "your goal is to retire yourself"
- throwaway2037 11mo ago
  What is wrong with emojis in comments? I see no issue with it. Do I do it myself? No. Would I pushback if a young person added emojis to comments? No. I am looking at "the content, not the colour".
  - chihuahua 11mo ago
    I think GP may be thinking that emojis in PR comments (plus the other red flags they mentioned) are the result of copy/paste from LLM output, which might imply that the person who does mindless copy/pasting is not adding anything and could be replaced by LLM automation.
  - venturecruelty 11mo ago
    The point is that heavy emoji use means AI was likely used to produce a changeset, not that emojis are inherently bad.
  - icedchai 11mo ago
    The emojis are not a problem themselves. They're a warning sign: slop is (probably) present, look deeper.
bryanrasmussen 11mo ago
having dealt with a few people who just copy/pasted Stackoverflow I really feel that using an LLM is an improvement. That is at least for the people who don't understand what they're doing, the LLM tends to come out with something I can at least turn into something useful. It might be reversed though for people who know what they're doing. IF they know what they're doing they might theoretically be able to put together some stackoverflow results that make sense, and build something up from that better than what gets generated from LLM (I am not asserting this would happen, and thinking it might be the case) However I don't know as I've never known anyone who knew what they were doing who also just copy/pasted some stackoverflow or delegated to LLM significantly.
water-data-dude 11mo ago
Is/was copy/pasting from Stackoverflow considered harmful? You have a problem, you do a web search and you find someone who asked the same question on SO, and there's often a solution. You might be specifically talking about people who copy/paste without understanding, but I think it's still OK-ish to do that, since you can't make an entire [whatever you're coding up] by copy/pasting snippets from SO like you're cutting words out of a magazine for a ransom note. There's still thought involved, so it's more like training wheels that you eventually outgrow as you get more understanding.
- vkou 11mo ago
  > Is/was copy/pasting from Stackoverflow considered harmful? It at least forces you to tinker with whatever you copied over.
gunsch 11mo ago
Pair programming! Get hands-on with your junior engineers and their development process. Push them to think through things and not just ask the LLM everything.
- johnnyanmac 11mo ago
  I've seen some overly excessive pair programming initiatives out there, but it does baffle me why less people who struggle with this do it. Take even just 30 minutes to pair program on a problem and see their process and you can reveal so much. But I suppose my question is rhetorical. We're laying off hundreds of thousands of engineers and maming existing ones do the work of 3-4 engineers. Not much time to help the juniors.

anal_reactor 11mo ago

> This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested." I've learnt that saying this exact phrase does wonders when it comes to advancing your career. I used to argue against stupid ideas but not only did I achieve nothing, but I was also labelled uncooperative and technically incompetent. Then I became a "yes-man" and all problems went away.

throwaway2037 11mo ago
This. May you have great success! My PR comments that I get are so dumb. I can put the most obvious bugs in my code, but people are focused in the colour of the bike shed. I am happy to repaint the bike shed whatever colour they need it to be!
shagie 11mo ago
I was attempting to mock Claude's "You are absolutely right" style of response when corrected. I have seen responses to PRs that appear to be a copy and paste of my feedback into it and a copy and paste of the response and fixes into the PR. It may be the that the developer is incorporating the mannerisms of Claude into their own speech... that would be something to delve into (that was intentional). However, more often than not in today's world of software development such responses are more likely to indicate a copy and paste of LLM generated content.
- anal_reactor 11mo ago
  > However, more often than not in today's world of software development such responses are more likely to indicate a copy and paste of LLM generated content. This is nothing new. People rarely have independent thoughts, usually they just parrot whatever they've been told to parrot. LLMs created common world-wide standard on this parroting, which makes the phenomenon more evident, but it doesn't change the fact that it existed before LLMs. Have you ever had a conversation with an intelligent person and thought "wow that's refreshing"? Yeah. There's a reason why it feels so good.

johnnyanmac 11mo ago

I must ask once again why we are having these 5+ round interview cycles and we aren't able to filter for qualities that the work requires of its talent. What are all those rounds for if we're getting engineers who aren't as valued for the team's needs at the end of the pipeline?

venturecruelty 11mo ago
It's the cargo cult kayfabe of it all. People do it because Google used to do it, now it's just spread like a folk religion. But nobody wants guilds or licensure, so we have to make everyone do a week-long take-home and then FizzBuzz in front of a very awkward committee. Might as well just read chicken bones, at least that would be less humiliating.
- johnnyanmac 11mo ago
  Yeah, I'd be more than fine with licensing if I didn't have to keep going through 5 rounds of trivia only to be ghosted. Let me do that once and show I can code my way out of a paper bag.
- nradov 11mo ago
  And who would write the guild membership or licensure criteria? How much should those focus on ReactJS versus validation criteria for cruise missile flight control software?
  - throwup238 11mo ago
    Guild members? Who else? You’re asking these rhetorical questions as if we haven’t had centuries of precedent here, both bad and good. How does the AMA balance between neurosurgeons and optometrists? Bar associations between corporate litigators and family estate lawyers? Professional engineering associations between civil engineers and chemical engineers?
    - nradov 11mo ago

locknitpicker 11mo ago

> Part of the challenge (and I don't have an answer either) is there are some juniors who use AI to assist... and some who use it to delegate all of their work to. This is not limited to junior devs. I had the displeasure of working with a guy who was hired as a senior dev who heavily delegated any work they did. He failed to even do the faintest review of what the coding agent and of course did zero testing. At one time these stunts resulted in a major incident where one of these glorious PRs pushed code that completely inverted a key business rule and resulted in paying customers being denied access to a paid product. Sometimes people are slackers with little to no ownership or pride in their craftsmanship, and just stumbled upon a career path they are not very good at. They start at juniors but they can idle long enough to waddle their way to senior positions. This is not a LLM problem, or caused by it.

sevenseacat 11mo ago

> This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested." And then in the next PR, you have to request the exact same changes