4 ms·

How are people collaborating on code when using AI tools to generate patches? We hold code review dear as a tool to make sure more than one set of eyeballs has

How are people collaborating on code when using AI tools to generate patches? We hold code review dear as a tool to make sure more than one set of eyeballs has been over a change before it goes into production, and more than one person has the context behind the code to be able to fix it in future. As model generated code becomes the norm I’m seeing code from junior engineers that they haven’t read and possible nor do they understand. For example, one Python script calling another using exec instead of importing it as a module, or writing code that is already available as a very common part of the standard library. In such cases, are we asking people to mark their code as auto generated? Should we review their prompts instead of the code? Should we require the prompt to code step be deterministic? Should we see their entire prompt context and not just the prompt they used to build the finished patch? I feel like a lot of the value of code review is to bring junior engineers up to higher levels. To that extent each review feels like an end of week school test, and I’m getting handed plagiarised AI slop to mark instead of something that maps properly to what the student does or does not know. Pair programming is another great teaching tool. Soon, it might be the only one left.

nicoburns 2y ago
That level of submission sounds like a "reject on sight, go back and do it again without Ai, if you need help then ask" to me.
- silvestrov 2y ago
  Then you get pushback from management: we don't have time for the perfect solution, if it works, it's good.
  - scottlamb 2y ago
    Pathologically short-term thinking from management is a problem that far predates AI (computers, even), and the solution hasn't changed: go somewhere better and watch the old place burn.
    - oneshtein 2y ago
      Can we replace management with AI?
      - FridgeSeal 2y ago
        We all keep making this joke, and one day someone’s going to try it. I have a feeling it’ll be moderately successful too.
      - p_l 2y ago
        I heard of multiple "let's make an AI to show who to fire to increase efficiency" get canceled after inevitably the AI points towards upper management which then cans the project because they can't spin it to have themselves excluded from analysis.
      - D-Coder 2y ago
        Not yet. Give it another year.
- danielbln 2y ago
  They can pull the code out of a magical oracle somewhere in the misty mountains for all I care, as long as they can telle what the code does and why it was placed there. If they can't, it's rejected. Use whatever tool you like, but the buck stops with you.
  - gorgoiler 2y ago
    My problem is a co worker has put an OK-ish wooden toy train in front of me presenting it as their own work, so I take some time to show them how to hold the plane for a better finish on the grain, how to pick timber so that the knots are faced inward, and how contrasting stock takes up wood stain in different way highlighting the elements of the piece. They nod politely, go back to Ali Express and pick a slightly different toy train to present as their own, commit it, and move on. Too romantic? Quite probably.
    - danielbln 2y ago
      At some point that employee has to be shown the door, because you can also order from Ali Express yourself, you don't need a well paid employee to do that that doesn't provide additional value.
      - pjmlp 2y ago
        Someone has yet to discover how offshoring works in practice. This is only one step further. Management not only doesn't care, they welcome the employee doing their Ali Express purchase, while saving their own operational costs.
cube2222 2y ago
They are collaborating by making sure they’ve read and understood the code that was generated, and usually edited it, too. Sometimes, when e.g. making an ad-hoc visualization or debugging tool, one may emphasize that they just had AI generate it and didn’t read into the details. Occasionally it makes sense. But if someone is making PRs without disclosing their lack of understanding of them because of generating most of it and not making sure everything is correct, that seems like a cultural issue, primarily. But I suppose you can start having such people walk you through their PRs, which should at least reveal their lack of understanding, if it’s in fact the case. Point is, this is imo not an experience inherent to LLM usage, and there’s also not much point reviewing LLM prompts because of the strong non-determinism involved.
lionkor 2y ago
AI is a tool, not a solution. If someone uses it to write code they don't understand and that is flawed, that should never pass code review. If your AI generated code passes code review without any questions asked and without any hints that it's AI generated (or AI was used in some way), then it doesnt matter that it was used. The person submitting the code is still responsible, and the reviewer is equally responsible. You typically have tests to make sure it behaved correctly, too.
- 2mol 2y ago
  That's beside the point, we all know that one terrible engineer who is just incredibly productive and completely lacks any self-critical instincts. With LLMs this type of output becomes much more easy to produce, so what are you going to do if you're asked to review 10x the amount of code that you're reviewing right now?
  - tylersmith 2y ago
    Fire the bad engineer.
  - lionkor 2y ago
    Something similar happened in my job. I reported the over-engineered, completely senseless solution, after the "dev" got really angry at me about simply reviewing his code. He was gone the next week. If you have a boss, tell him when this happens. If you don't, fire him/her yourself. If the amount of complexity introduced is not worth the use cases it covers, in a business sense, then that's a review that rejects the code on that basis.
  - bigfatkitten 2y ago
    If they're a net negative to the team, their manager needs to have a performance conversation with them.
linsomniac 2y ago
I do believe my code review of a junior developer submitting "AI slop without reviewing or understanding it" would be fairly blunt: "If you are just taking AI generated content verbatim without code reviewing it and understanding it, you are providing no value over just having the AI do the work directly. If you are providing no value over AI doing your job, the company can replace you with AI." I'm wondering if the junior employee just isn't educated enough to be using the AI, like they don't know Python at all so they don't understand the difference between "exec" and "import"? But I'm also inclined to think that they should be using it to learn, if the goal of the company and the employee is to move from junior to mid to senior. Like ask another model "can you improve this code", "does this code make sense", "can you explain this code to me", and anything in the code you don't understand, research and learn. But, there are employees who are just going to grind it out, doing "just enough" to get something working, and it can at times be hard to figure out why they are there and why they aren't working towards more. It could be as complex is physical or mental struggles that put "just enough to get it working" at the far edge of their abilities, maybe that literally is the most they can do. How do you learn that and adjust so they can excel within their constraints?
righthand 2y ago
Anecdotally, I frequently dig into source code where the stack trace points me, look up functions, debug in local envs, etc. meanwhile my coworker is working on the same problem and talking to an LLM and I often get to the solution before he does. I don't think I've had my John Henry moment quite yet.

nicoburns 2y ago

That level of submission sounds like a "reject on sight, go back and do it again without Ai, if you need help then ask" to me.

silvestrov 2y ago
Then you get pushback from management: we don't have time for the perfect solution, if it works, it's good.
- scottlamb 2y ago
  Pathologically short-term thinking from management is a problem that far predates AI (computers, even), and the solution hasn't changed: go somewhere better and watch the old place burn.
  - oneshtein 2y ago
    Can we replace management with AI?
    - FridgeSeal 2y ago
      We all keep making this joke, and one day someone’s going to try it. I have a feeling it’ll be moderately successful too.
    - p_l 2y ago
      I heard of multiple "let's make an AI to show who to fire to increase efficiency" get canceled after inevitably the AI points towards upper management which then cans the project because they can't spin it to have themselves excluded from analysis.
    - D-Coder 2y ago
      Not yet. Give it another year.
danielbln 2y ago
They can pull the code out of a magical oracle somewhere in the misty mountains for all I care, as long as they can telle what the code does and why it was placed there. If they can't, it's rejected. Use whatever tool you like, but the buck stops with you.
- gorgoiler 2y ago
  My problem is a co worker has put an OK-ish wooden toy train in front of me presenting it as their own work, so I take some time to show them how to hold the plane for a better finish on the grain, how to pick timber so that the knots are faced inward, and how contrasting stock takes up wood stain in different way highlighting the elements of the piece. They nod politely, go back to Ali Express and pick a slightly different toy train to present as their own, commit it, and move on. Too romantic? Quite probably.
  - danielbln 2y ago
    At some point that employee has to be shown the door, because you can also order from Ali Express yourself, you don't need a well paid employee to do that that doesn't provide additional value.
    - pjmlp 2y ago
      Someone has yet to discover how offshoring works in practice. This is only one step further. Management not only doesn't care, they welcome the employee doing their Ali Express purchase, while saving their own operational costs.

cube2222 2y ago

They are collaborating by making sure they’ve read and understood the code that was generated, and usually edited it, too. Sometimes, when e.g. making an ad-hoc visualization or debugging tool, one may emphasize that they just had AI generate it and didn’t read into the details. Occasionally it makes sense. But if someone is making PRs without disclosing their lack of understanding of them because of generating most of it and not making sure everything is correct, that seems like a cultural issue, primarily. But I suppose you can start having such people walk you through their PRs, which should at least reveal their lack of understanding, if it’s in fact the case. Point is, this is imo not an experience inherent to LLM usage, and there’s also not much point reviewing LLM prompts because of the strong non-determinism involved.

lionkor 2y ago

AI is a tool, not a solution. If someone uses it to write code they don't understand and that is flawed, that should never pass code review. If your AI generated code passes code review without any questions asked and without any hints that it's AI generated (or AI was used in some way), then it doesnt matter that it was used. The person submitting the code is still responsible, and the reviewer is equally responsible. You typically have tests to make sure it behaved correctly, too.

2mol 2y ago
That's beside the point, we all know that one terrible engineer who is just incredibly productive and completely lacks any self-critical instincts. With LLMs this type of output becomes much more easy to produce, so what are you going to do if you're asked to review 10x the amount of code that you're reviewing right now?
- tylersmith 2y ago
  Fire the bad engineer.
- lionkor 2y ago
  Something similar happened in my job. I reported the over-engineered, completely senseless solution, after the "dev" got really angry at me about simply reviewing his code. He was gone the next week. If you have a boss, tell him when this happens. If you don't, fire him/her yourself. If the amount of complexity introduced is not worth the use cases it covers, in a business sense, then that's a review that rejects the code on that basis.
- bigfatkitten 2y ago
  If they're a net negative to the team, their manager needs to have a performance conversation with them.

linsomniac 2y ago

I do believe my code review of a junior developer submitting "AI slop without reviewing or understanding it" would be fairly blunt: "If you are just taking AI generated content verbatim without code reviewing it and understanding it, you are providing no value over just having the AI do the work directly. If you are providing no value over AI doing your job, the company can replace you with AI." I'm wondering if the junior employee just isn't educated enough to be using the AI, like they don't know Python at all so they don't understand the difference between "exec" and "import"? But I'm also inclined to think that they should be using it to learn, if the goal of the company and the employee is to move from junior to mid to senior. Like ask another model "can you improve this code", "does this code make sense", "can you explain this code to me", and anything in the code you don't understand, research and learn. But, there are employees who are just going to grind it out, doing "just enough" to get something working, and it can at times be hard to figure out why they are there and why they aren't working towards more. It could be as complex is physical or mental struggles that put "just enough to get it working" at the far edge of their abilities, maybe that literally is the most they can do. How do you learn that and adjust so they can excel within their constraints?

righthand 2y ago

Anecdotally, I frequently dig into source code where the stack trace points me, look up functions, debug in local envs, etc. meanwhile my coworker is working on the same problem and talking to an LLM and I often get to the solution before he does. I don't think I've had my John Henry moment quite yet.