5 ms·

I think a lot of people would disagree with this article on HN, but I’ve yet to see too many people say it’s made their coworkers more productive. That is, do p

I think a lot of people would disagree with this article on HN, but I’ve yet to see too many people say it’s made their coworkers more productive. That is, do people feel like they’re getting better, more reviewable PRs? Personally, I’ve been seeing the number of changes for a PR starting to reach into the mid-hundreds now. And fundamentally the developers who make them don’t understand how they work. They often think they do, but then I’ll ask them something about the design and they’ll reply: “IDK Claude did that.” By no means am I down on AI, but I think proper procedures need to be put into place unless we want a giant bomb in our code base.

aantix 1y ago
I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. AI may be multi-threaded, but there's still a human, global interpreter lock in place. :D If you put the code up for review, regardless of the source, you should fundamentally understand how it works. This raises a broader point about AI and productivity: while AI promises parallelism, there's still the human in the middle who is responsible for the code. The promise of "parallelism" is overstated. 100's of PRs should not be trusted. Or at least not without the c-suite understanding such risks. Maybe you're a small startup looking to get out the door as quickly as possible, so.. YOLO. But it's going to be a hot mess. A "clean up in aisle nine" level mess.
- ryandrake 1y ago
  It's insane that any company would just be OK with "IDK Claude did that" any more than a 2010 version of that company would be OK with "IDK I copy pasted from StackOverflow." Have engineering managers actually drank this Kool-aid to the point where they're actually OK with their direct reports just chucking PRs over the wall that they don't even understand?
  - pinoy420 1y ago
    [dead]
  - throwawaysleep 1y ago
    Depends on your incentives. People anecdotally seem far more impressed with buggy stuff shipped fast than good stuff shipped slowly. Lots of companies just accept bugs as something that happens.
    - dontlikeyoueith 1y ago
      Depends on your problem space too. Calendar app for local social clubs? Ship it and fix it later. B2B payments software that triggers funds transfers? JFC I hope you PIP people for that.
  - Imustaskforhelp 1y ago
    It is even more funnier when you realize that because Claude and all AI models are trained on data including stackoverflow. So I guess if you asked Claude why it did that, the truth of it might be "IDK I copy pasted from StackOverflow" The same stuff pasted with a different sticker. Looks good to me.
    - jaredcwhite 1y ago
      Ha, I genuinely laughed at that. Thank you!
  - liveoneggs 1y ago
    Of course they are okay with it. They changed the job function to be just that with forced(!) AI adoption.
  - f1shy 1y ago
    This is exactly how I see it. Is not about the tool, is how it is used. In 1990 that would have been “IDK I got it from a BBS” and in 1980 “got if from a magazine“. It doesn’t matter how you get there, you have to understand it. BTW I had a similar problem as I was manager in HW development, where the value of a resistor had no documented calculation. I would ask: where does it came from? If the answer was “I tried and it worked”, or “tested in lab until I found it” or in the 2000 “I run many simulations and was the best value” I would reject and ask for proper calculations, with WCA.
    - Andrex 1y ago
      As vibe coding becomes more commonplace you'll see these historical safeguards erode. That is the danger IMO. You're right, saying you got something off SO would get you laughed out of programming circles back in the day. We should be applying the same shame to people who vibe code, not encourage it, if we want human-parseable and maintainable software.
      - vkou 1y ago
        > That is the danger IMO. For whom is this a danger for? If we're paid to dig ditches and fill them, who are we to question our supreme leaders? They control the purse strings, so of course they know best.
        nemesisithetic 1y ago
        > If we're paid to dig ditches and fill them This is a very cruel punishment sometimes used in forced labor camps. You are describing torture.
  - j45 1y ago
    The same developers submitting Claude submissions can take 1-2 minutes asking for an explanation of what they're submitting and how it works. Might even learn. Stack Overflow had enough provenance of copying and pasting. Models may not. Provenance remains a thing or it can add risk to the code.
  - joseda-hg 1y ago
    I don't think it's common, but I've definitely seen it I've also seen "Ask ChatGPT if you're doing X right?", and basically signing off whatever it recommends without checking At this point I'm pretty confident I could trojan horse whatever decision I want from certain people by sending enough screenshots of ChatGPT agreeing with me
  - calebkaiser 1y ago
    I don't think this is an AI specific thing. I work in the field, and so I'm around some of the most enthusiastic adopters of LLMs, and from what I see, engineering cultures surrounding LLM usage typically match the org's previous general engineering culture. So, for example, by and large the orgs I've seen chucking Claude PRs over the wall with little review were previously chucking 100% human written PRs over the wall with little review. Similarly, the teams I see effectively using test suites to guide their code generation are the same teams that effectively use test suites to guide their general software engineering workflows.
    - throwanem 1y ago
      How long are you spending with a given team, and where per se in their "AI lifecycle?" I would expect (for example) a sales engineer to see this differently than a support engineer, if support engineers still existed.
  - siva7 1y ago
    If it pushes some nice velocity metric, most managers would be ok. Though you have to word it a bit differently of course.
  - throwanem 1y ago
    "Look, the build is green and CI belongs to another team, how perfectionist do you need us to be about this?" is the sort of response I would generally expect, and also in the case where AI was used.
  - danielmarkbruce 1y ago
    What about "the compiler did that" ?
- SoftTalker 1y ago
  As someone who doesn't use AI for writing code, why can't you just ask Claude to write up an explanation of each change for code review? Then at least you can look at whether the explanation seems sane.
  - threetonesun 1y ago
    It will fairly confidently state changes are "correct" for whatever reason it makes up. This becomes more of an issue with things that might be edge cases or vague requirements, in which case it's better to have AI write tests instead of the code.
  - ahoef 1y ago
    Claude also doesn't know, because Claude dreamt up changes that didn't work, then "fixed" them, "fixed" them again and in the process left swathes of code that isn't reached.
  - thegeomaster 1y ago
    This can be dangerous, because Claude doesn't truly understand why it did something. Whatever it writes a post-hoc justification which may or may not be accurate to the "intent". This is because these are still autoregressive models --- they have only the context to go on, not prior intent.
    - zahlman 1y ago
      Indeed. Watching it (well, Anthropic, really) cheat at Baba Is You and then try to give a rationalization for how it came up with the solution (qv. https://news.ycombinator.com/item?id=44473615 https://news.ycombinator.com/item?id=44473615) is quite instructive.
  - zahlman 1y ago
    Because the explanations will often not be sane; when they are sane, they will focus on irrelevant details and be maddeningly padded out unless you put inordinate effort into trying to control the AI's writing style. Ask pretty much any FOSS developer who has received AI-generated (both code and explanations) PRs on GitHub (and when you complain about these, the author will almost always use the same AI to generate responses) about their experiences. It's a huge time sink if you don't cut them off. There are plenty of projects out there now that have explicit policy documentation against such submissions and even boilerplate messages for rejecting them.
- NegativeLatency 1y ago
  > I don't think "IDK Claude did that" is a valid excuse. It's not, and yet I have seen that offered as an excuse several times.
  - grogenaut 1y ago
    Did you push back?
- f1shy 1y ago
  At least I would not accept from my team. Is borderline infuriating. And I would promptly insinuate, if that is the answer, next time I do not need you, I will ask directly Claude, you can stay home!
  - Herring 1y ago
    Maybe check what's their workload otherwise. Most engineers I've worked with want to do a good job and ship something useful. It's possible they're offloading work to the LLM because they're under a lot of pressure. (And in this case you can't make them stay home)
- corytheboyd 1y ago
  > The promise of "parallelism" is overstated. 100% my takeaway after trying to parallelize using worktrees. While Claude has no problem managing more than one context instance, I sure as hell do. It’s exhausting, to the point of slowing me down.
  - vouaobrasil 1y ago
    That's an intended effect. It doesn't matter to those in power who know what AI is really for. Once you get so exhausted that you can't work any more, there will be a hundred bright-eyed naïve programmers who will step into your place and who think they can do better. Until they burn out in a few years time.
    - corytheboyd 1y ago
      I have been wondering when I would start to feel aged out of the tech industry… gosh is it here already?
      - vouaobrasil 1y ago
        I don't know if it is but I'm certainly glad I left tech a long time ago...
- ToucanLoucan 1y ago
  > If you put the code up for review, regardless of the source, you should fundamentally understand how it works. Inb4 the chorus of whining from AI hypists accusing you of being an coastal elitist intellectual jerk for daring to ask that they might want to LEARN something. I am so over this anti-intellectual garbage. It's gotten to such a ridiculous place in our society and is literally going to get tons of people killed.
  - zahlman 1y ago
    I understand and agree with your frustration, but this is not what discourse here is supposed to look like.
- j-bos 1y ago
  > I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. I strongly agree, however manager^x do not and want see report the massive "productivity" gains.
  - Izikiel43 1y ago
    You tell them clippy’s revengeance pr caused an outage worth millions of dollars because of push for productivity and they shouldn’t bother you for a couple of months.
  - dingnuts 1y ago
    Sane CTOs think "Claude did that" is invalid. I assure you: those leaders exist. Refuse to work for idiots who think bots can be held accountable. You must must understand every line of code yourself. "Claude did that" is functionally equivalent to "idk I copied that from r/programming" and is totally unacceptable for a professional
    - derf_ 1y ago
      > You must must understand every line of code yourself. I have never seen this standard reached for any real codebase of any size. Even in projects with a reputation for a strong review culture, people know who the "easy" reviewers are and target them for the dicey stuff (they are often the most overloaded... which only causes them to get more overloaded). I've seen people explicitly state they are just "rubber stamping" PRs. Literally no one reviews every line of third-party dependencies, and especially not when they are updated routinely. I've seen over a million lines of security-sensitive third-party code integrated and pushed out to hundreds of millions of users by a handful of developers in a matter of months. I've seen developers write their new green-field code as a third-party library to circumvent the review process that would have been applied if it had been developed as a series of first-party PRs. None of that had anything to do with AI. It all predated AI coding tools. That is how humans behave. Does this create ticking time-bombs? It absolutely does. You do the best you can. You triage and deal with the most important things according to your best judgment, and circle back to the rest as time and attention allow. If your judgment is good, it's mostly okay. Some day it might not be. But I do not think that you can argue that the optimal level of risk is zero, outside of a few specialized contexts like space shuttles and nuclear reactors. I know. It hurts my soul, too. But reality isn't pretty, and worse is better.
- kibwen 1y ago
  > I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. That will work, but only until the people filing these PRs go crying to their managers that you refuse to merge any of their code, at which point you'll be given a stern reprimand from your betters to stop being so picky. Have fun vibe-reviewing.
- pkdpic 1y ago
  Yeah in a perfect world I absolutely agree. But the reality I'm observing is that everything continues to always be behind schedule (not a new phenomena) and if anything expectations from project leads / management are just getting less realistic leaving Jr devs in even more of a position of "no time to be curious or learn deeply just get it done" and teamleads reviewing PRs in a maybe even worse position of "no time to get into a deep review / mentorship session just figure out if it breaks anything or not". And ultimately clients are still just living in fantasy land in terms of expectations both in terms of build out time for basic features / patches and also how much AI razzle-dazzle they expect for new project proposals. Nothing can move fast enough to keep up with these hype-fueled TED talk expectations all the way up the chain. I don't know if there's any solution and I'm sure it's not like this everywhere but I'm also sure I'm not alone. At this point I'm just trying to keep my feet wet on "AI" related projects until the hype dust settles so I can reassess what this industry even is anymore. Maybe it's not too late to get a single subject credential and go teach math or finger painting or something.
throwawaysleep 1y ago
A lot depends heavily on how you define productive. At one of my jobs, the PRs are far less reviewable, but now the devs write tests when they didn’t used to bother. I’ve always viewed reviewing PRs as work without recognition, so I never did much of it anyway, but but now that there are passing tests, I often approve without more than a cursory skim. So yes, it has made their work more productive for me to get off my plane.
- skydhash 1y ago
  But are the test actually useful? You can have a test suite that is actually harmful if it’s not ensuring business rules or domain correctness. Anything else makes for a brittle dev ecosystem.
- teiferer 1y ago
  Thanks for this take. It perfectly illustrates the productivity illusion. Mostly blindly approving PRs mainly because they now contain some form of equally AI-generated tests is exactly the producticity fallacy that's the issue here.
whstl 1y ago
> That is, do people feel like they’re getting better, more reviewable PRs? No, like you I’m getting more PRs that are less reviewable. It multiplies what you’re capable of. So you’ll get a LOT of low quality code from devs who aren’t much into quality.
- TheSamFischer 1y ago
  [dead]
alexander2002 1y ago
AI is like a giant calculator you need a formula to make it work for your usecase
- asciii 1y ago
  That's a great analogy. I recently read about integrating AI similar to the use of calculators in math class -- learn how to do the basic operations first +,-,/,* and then use calculator to scale so you get some theoretical grounding
- SamInTheShell 1y ago
  Accurate. It's autocomplete on steroids.
- liveoneggs 1y ago
  Calculators are mostly deterministic and AI is explicitly not. When computers give different answers to the same questions it's a fundamental shift in how we work with them.
uludag 1y ago
I've been at the same company both before and after the AI revolution. I've felt something similar. People seem to be more detached, more aloof in their work. I feel like we're discussing our code less and are less able to have coherent big-picture plans concerning the code at-large.
- deleted 1y ago
  [deleted]
- deleted 1y ago
  [deleted]
- vouaobrasil 1y ago
  The ultimate purpose of AI is indeed to remove cognitive autonomy. Little pieces here and there may seem empowering, but added all up, it pretty much takes control away from people.
  - nemesisithetic 1y ago
    Bingo. This is a drive to go back to the good old days of leasing mainframes, before the peasants got their grubby hands on the means of computation.
qsort 1y ago
It's extremely hard to measure productivity correctly and self-reports are worthless. I don't think AI tools are a net negative in the average case (people are definitely indexing too much on that goddamn METR article) but "i'm 10x more productive, source trust me bro" is equally nonsense. Using AI tooling means, at least in part, betting on the future.
- didericis 1y ago
  > Using AI tooling means, at least in part, betting on the future. It means betting on a particular LLM centric vision of the future. I’m still agnostic on that. I think LLMs allow for the creation of a lot of one off scripts and things for people that wouldn’t otherwise be coding, but I have yet to be convinced that more AI usage in a sufficiently senior software development team is more valuable than the traditional way of doing things. I think there’s a fundamental necessity for a human to articulate what a given piece of software should do with a high level of specificity that can’t ever be avoided. The best you can do is piggy back off of higher level language and abstractions that guess what the specifics should be, but I don’t think it’s realistic to think all combinations of all business logic and ui can be boiled down to common patterns that an LLM could infer. And even if that were true, people get bored/like novelty enough that they’ll always want new human created stuff to shove into the training set.
  - danielmarkbruce 1y ago
    An LLM is not a tool to allow you to add a layer of abstraction. It's a worker.
    - didericis 1y ago
      It works by translating language abstractions to code. The probability of plain english being correctly translated to code depends on existing code and documented abstractions describing lower level functionality.
      - danielmarkbruce 1y ago
pfisherman 1y ago
> Personally, I’ve been seeing the number of changes for a PR starting to reach into the mid-hundreds now. And fundamentally the developers who make them don’t understand how they work. Could this be fixed by adjusting how tickets are scoped?
chocolatemario 1y ago
This just sounds like a low quality bar or PRs that are too wide in large in scope for AI. In any case, it sounds like taking a single stab at it with an LLM and calling it good. I’m not really AI for everything type of dev, but there are some tasks that AI excels at. If you’re doing it for feature work on a high churn product without a tight grip on the reins, I fear for your products future.
ratelimitsteve 1y ago
>I’ll ask them something about the design and they’ll reply: “IDK Claude did that.” I would want someone entirely off of my team if they did that. Anyone who pushes code they don't understand at least well enough to answer "What does that do?" and "Why did you do it that way?" deserves for their PR to be flat out rejected in whole, not just altered.
rafaelmn 1y ago
I've seen the worst impact on mid/junior level devs. Where they would have to struggle through a problem before - AI is like a magic shortcut to something that looks like it works. And then they submit that crap and I can't trust them again - they will give me AI code without even understanding what it does fully. It robbed them of the learning process and made them even less useful, while making it seem to them they were achieving something. I'm seeing these kinds of people getting removed from workforce fast - you can probably prompt the AI better on your own, have one less layer of indirection and it will be faster.
- danielmarkbruce 1y ago
  Exactly this. In the right hands the outputs from LLMs are amazing. Not always right, but can be checked. In the wrong hands they are worse than useless because they cannot be checked.
nlawalker 1y ago
> I’ll ask them something about the design and they’ll reply: “IDK Claude did that.” "Then your job is to go ask Claude and get back to me. On that note, if that's what I'm paying you for now, I might be paying you too much..." I'm really interested to see how the intersection of AI and accountability develops over the next few years. It seems like everyone's primary job will effectively be taking accountability for the AI they're driving, and the pay will be a function of how much and what kind of accountability you're taking and the overall stakes.
- moritzwarhier 1y ago
  I hope so, in reality, people who review code are not always the superiors of the other party :) Not meaning to put on a holier-than-thou hat here. But I hated this many times already. E.g. having to approve code that contains useless comments that were clearly generated by AI, but on the other hand, it's nitpicking to bring it up during review... If my colleagues would do that more often and for more parts of the code, that's where I really start getting inclined to look for a new job.
SatvikBeri 1y ago
A new coworker solved a problem that we'd basically been unable to fix for 2 years, and we were relying on workarounds. This was especially impressive because it was totally outside his expertise. When I asked how he did it, a big part of his process was using LLMs. For the most part, I've noticed my LLM-using coworkers producing better PRs, but I haven't noticed anyone producing more.
- subarctic 1y ago
  That's the thing, there's a big difference between using AI to help you solve a problem and using it to write code that you don't understand (and not take the time to understand it). It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did. But these are powerful tools and they're not going away anytime soon
  - dragonwriter 1y ago
    > It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did. It does, but that experience was a real thing long before AI-generated code.
josefritzishere 1y ago
I too have seen anecdotes that people "feel" more productive. But science says that they're 19% less productive. I'm going to go with science on this one.
- vouaobrasil 1y ago
  Further, the problem with the word "productive" is that it's often measured in quantity and speed, not quality.
- davely 1y ago
  The study that said devs were 19% less productive was flawed in a number of ways: small sample size, poorly chosen tasks, no prior training on AI tools. The science is by no means conclusive. A joint Harvard / MIT study found developers completed tasks up to 50% faster [0]! [0] https://arxiv.org/abs/2302.06590 https://arxiv.org/abs/2302.06590

aantix 1y ago

I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. AI may be multi-threaded, but there's still a human, global interpreter lock in place. :D If you put the code up for review, regardless of the source, you should fundamentally understand how it works. This raises a broader point about AI and productivity: while AI promises parallelism, there's still the human in the middle who is responsible for the code. The promise of "parallelism" is overstated. 100's of PRs should not be trusted. Or at least not without the c-suite understanding such risks. Maybe you're a small startup looking to get out the door as quickly as possible, so.. YOLO. But it's going to be a hot mess. A "clean up in aisle nine" level mess.

ryandrake 1y ago
It's insane that any company would just be OK with "IDK Claude did that" any more than a 2010 version of that company would be OK with "IDK I copy pasted from StackOverflow." Have engineering managers actually drank this Kool-aid to the point where they're actually OK with their direct reports just chucking PRs over the wall that they don't even understand?
- pinoy420 1y ago
  [dead]
- throwawaysleep 1y ago
  Depends on your incentives. People anecdotally seem far more impressed with buggy stuff shipped fast than good stuff shipped slowly. Lots of companies just accept bugs as something that happens.
  - dontlikeyoueith 1y ago
    Depends on your problem space too. Calendar app for local social clubs? Ship it and fix it later. B2B payments software that triggers funds transfers? JFC I hope you PIP people for that.
- Imustaskforhelp 1y ago
  It is even more funnier when you realize that because Claude and all AI models are trained on data including stackoverflow. So I guess if you asked Claude why it did that, the truth of it might be "IDK I copy pasted from StackOverflow" The same stuff pasted with a different sticker. Looks good to me.
  - jaredcwhite 1y ago
    Ha, I genuinely laughed at that. Thank you!
- liveoneggs 1y ago
  Of course they are okay with it. They changed the job function to be just that with forced(!) AI adoption.
- f1shy 1y ago
  This is exactly how I see it. Is not about the tool, is how it is used. In 1990 that would have been “IDK I got it from a BBS” and in 1980 “got if from a magazine“. It doesn’t matter how you get there, you have to understand it. BTW I had a similar problem as I was manager in HW development, where the value of a resistor had no documented calculation. I would ask: where does it came from? If the answer was “I tried and it worked”, or “tested in lab until I found it” or in the 2000 “I run many simulations and was the best value” I would reject and ask for proper calculations, with WCA.
  - Andrex 1y ago
    As vibe coding becomes more commonplace you'll see these historical safeguards erode. That is the danger IMO. You're right, saying you got something off SO would get you laughed out of programming circles back in the day. We should be applying the same shame to people who vibe code, not encourage it, if we want human-parseable and maintainable software.
    - vkou 1y ago
      > That is the danger IMO. For whom is this a danger for? If we're paid to dig ditches and fill them, who are we to question our supreme leaders? They control the purse strings, so of course they know best.
      - nemesisithetic 1y ago
        > If we're paid to dig ditches and fill them This is a very cruel punishment sometimes used in forced labor camps. You are describing torture.
- j45 1y ago
  The same developers submitting Claude submissions can take 1-2 minutes asking for an explanation of what they're submitting and how it works. Might even learn. Stack Overflow had enough provenance of copying and pasting. Models may not. Provenance remains a thing or it can add risk to the code.
- joseda-hg 1y ago
  I don't think it's common, but I've definitely seen it I've also seen "Ask ChatGPT if you're doing X right?", and basically signing off whatever it recommends without checking At this point I'm pretty confident I could trojan horse whatever decision I want from certain people by sending enough screenshots of ChatGPT agreeing with me
- calebkaiser 1y ago
  I don't think this is an AI specific thing. I work in the field, and so I'm around some of the most enthusiastic adopters of LLMs, and from what I see, engineering cultures surrounding LLM usage typically match the org's previous general engineering culture. So, for example, by and large the orgs I've seen chucking Claude PRs over the wall with little review were previously chucking 100% human written PRs over the wall with little review. Similarly, the teams I see effectively using test suites to guide their code generation are the same teams that effectively use test suites to guide their general software engineering workflows.
  - throwanem 1y ago
    How long are you spending with a given team, and where per se in their "AI lifecycle?" I would expect (for example) a sales engineer to see this differently than a support engineer, if support engineers still existed.
- siva7 1y ago
  If it pushes some nice velocity metric, most managers would be ok. Though you have to word it a bit differently of course.
- throwanem 1y ago
  "Look, the build is green and CI belongs to another team, how perfectionist do you need us to be about this?" is the sort of response I would generally expect, and also in the case where AI was used.
- danielmarkbruce 1y ago
  What about "the compiler did that" ?
SoftTalker 1y ago
As someone who doesn't use AI for writing code, why can't you just ask Claude to write up an explanation of each change for code review? Then at least you can look at whether the explanation seems sane.
- threetonesun 1y ago
  It will fairly confidently state changes are "correct" for whatever reason it makes up. This becomes more of an issue with things that might be edge cases or vague requirements, in which case it's better to have AI write tests instead of the code.
- ahoef 1y ago
  Claude also doesn't know, because Claude dreamt up changes that didn't work, then "fixed" them, "fixed" them again and in the process left swathes of code that isn't reached.
- thegeomaster 1y ago
  This can be dangerous, because Claude doesn't truly understand why it did something. Whatever it writes a post-hoc justification which may or may not be accurate to the "intent". This is because these are still autoregressive models --- they have only the context to go on, not prior intent.
  - zahlman 1y ago
    Indeed. Watching it (well, Anthropic, really) cheat at Baba Is You and then try to give a rationalization for how it came up with the solution (qv. https://news.ycombinator.com/item?id=44473615 https://news.ycombinator.com/item?id=44473615) is quite instructive.
- zahlman 1y ago
  Because the explanations will often not be sane; when they are sane, they will focus on irrelevant details and be maddeningly padded out unless you put inordinate effort into trying to control the AI's writing style. Ask pretty much any FOSS developer who has received AI-generated (both code and explanations) PRs on GitHub (and when you complain about these, the author will almost always use the same AI to generate responses) about their experiences. It's a huge time sink if you don't cut them off. There are plenty of projects out there now that have explicit policy documentation against such submissions and even boilerplate messages for rejecting them.
NegativeLatency 1y ago
> I don't think "IDK Claude did that" is a valid excuse. It's not, and yet I have seen that offered as an excuse several times.
- grogenaut 1y ago
  Did you push back?
f1shy 1y ago
At least I would not accept from my team. Is borderline infuriating. And I would promptly insinuate, if that is the answer, next time I do not need you, I will ask directly Claude, you can stay home!
- Herring 1y ago
  Maybe check what's their workload otherwise. Most engineers I've worked with want to do a good job and ship something useful. It's possible they're offloading work to the LLM because they're under a lot of pressure. (And in this case you can't make them stay home)
corytheboyd 1y ago
> The promise of "parallelism" is overstated. 100% my takeaway after trying to parallelize using worktrees. While Claude has no problem managing more than one context instance, I sure as hell do. It’s exhausting, to the point of slowing me down.
- vouaobrasil 1y ago
  That's an intended effect. It doesn't matter to those in power who know what AI is really for. Once you get so exhausted that you can't work any more, there will be a hundred bright-eyed naïve programmers who will step into your place and who think they can do better. Until they burn out in a few years time.
  - corytheboyd 1y ago
    I have been wondering when I would start to feel aged out of the tech industry… gosh is it here already?
    - vouaobrasil 1y ago
      I don't know if it is but I'm certainly glad I left tech a long time ago...
ToucanLoucan 1y ago
> If you put the code up for review, regardless of the source, you should fundamentally understand how it works. Inb4 the chorus of whining from AI hypists accusing you of being an coastal elitist intellectual jerk for daring to ask that they might want to LEARN something. I am so over this anti-intellectual garbage. It's gotten to such a ridiculous place in our society and is literally going to get tons of people killed.
- zahlman 1y ago
  I understand and agree with your frustration, but this is not what discourse here is supposed to look like.
j-bos 1y ago
> I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. I strongly agree, however manager^x do not and want see report the massive "productivity" gains.
- Izikiel43 1y ago
  You tell them clippy’s revengeance pr caused an outage worth millions of dollars because of push for productivity and they shouldn’t bother you for a couple of months.
- dingnuts 1y ago
  Sane CTOs think "Claude did that" is invalid. I assure you: those leaders exist. Refuse to work for idiots who think bots can be held accountable. You must must understand every line of code yourself. "Claude did that" is functionally equivalent to "idk I copied that from r/programming" and is totally unacceptable for a professional
  - derf_ 1y ago
    > You must must understand every line of code yourself. I have never seen this standard reached for any real codebase of any size. Even in projects with a reputation for a strong review culture, people know who the "easy" reviewers are and target them for the dicey stuff (they are often the most overloaded... which only causes them to get more overloaded). I've seen people explicitly state they are just "rubber stamping" PRs. Literally no one reviews every line of third-party dependencies, and especially not when they are updated routinely. I've seen over a million lines of security-sensitive third-party code integrated and pushed out to hundreds of millions of users by a handful of developers in a matter of months. I've seen developers write their new green-field code as a third-party library to circumvent the review process that would have been applied if it had been developed as a series of first-party PRs. None of that had anything to do with AI. It all predated AI coding tools. That is how humans behave. Does this create ticking time-bombs? It absolutely does. You do the best you can. You triage and deal with the most important things according to your best judgment, and circle back to the rest as time and attention allow. If your judgment is good, it's mostly okay. Some day it might not be. But I do not think that you can argue that the optimal level of risk is zero, outside of a few specialized contexts like space shuttles and nuclear reactors. I know. It hurts my soul, too. But reality isn't pretty, and worse is better.
kibwen 1y ago
> I don't think "IDK Claude did that" is a valid excuse. Immediate rejection. That will work, but only until the people filing these PRs go crying to their managers that you refuse to merge any of their code, at which point you'll be given a stern reprimand from your betters to stop being so picky. Have fun vibe-reviewing.
pkdpic 1y ago
Yeah in a perfect world I absolutely agree. But the reality I'm observing is that everything continues to always be behind schedule (not a new phenomena) and if anything expectations from project leads / management are just getting less realistic leaving Jr devs in even more of a position of "no time to be curious or learn deeply just get it done" and teamleads reviewing PRs in a maybe even worse position of "no time to get into a deep review / mentorship session just figure out if it breaks anything or not". And ultimately clients are still just living in fantasy land in terms of expectations both in terms of build out time for basic features / patches and also how much AI razzle-dazzle they expect for new project proposals. Nothing can move fast enough to keep up with these hype-fueled TED talk expectations all the way up the chain. I don't know if there's any solution and I'm sure it's not like this everywhere but I'm also sure I'm not alone. At this point I'm just trying to keep my feet wet on "AI" related projects until the hype dust settles so I can reassess what this industry even is anymore. Maybe it's not too late to get a single subject credential and go teach math or finger painting or something.

throwawaysleep 1y ago

A lot depends heavily on how you define productive. At one of my jobs, the PRs are far less reviewable, but now the devs write tests when they didn’t used to bother. I’ve always viewed reviewing PRs as work without recognition, so I never did much of it anyway, but but now that there are passing tests, I often approve without more than a cursory skim. So yes, it has made their work more productive for me to get off my plane.

skydhash 1y ago
But are the test actually useful? You can have a test suite that is actually harmful if it’s not ensuring business rules or domain correctness. Anything else makes for a brittle dev ecosystem.
teiferer 1y ago
Thanks for this take. It perfectly illustrates the productivity illusion. Mostly blindly approving PRs mainly because they now contain some form of equally AI-generated tests is exactly the producticity fallacy that's the issue here.

whstl 1y ago

> That is, do people feel like they’re getting better, more reviewable PRs? No, like you I’m getting more PRs that are less reviewable. It multiplies what you’re capable of. So you’ll get a LOT of low quality code from devs who aren’t much into quality.

TheSamFischer 1y ago
[dead]

alexander2002 1y ago

AI is like a giant calculator you need a formula to make it work for your usecase

asciii 1y ago
That's a great analogy. I recently read about integrating AI similar to the use of calculators in math class -- learn how to do the basic operations first +,-,/,* and then use calculator to scale so you get some theoretical grounding
SamInTheShell 1y ago
Accurate. It's autocomplete on steroids.
liveoneggs 1y ago
Calculators are mostly deterministic and AI is explicitly not. When computers give different answers to the same questions it's a fundamental shift in how we work with them.

uludag 1y ago

I've been at the same company both before and after the AI revolution. I've felt something similar. People seem to be more detached, more aloof in their work. I feel like we're discussing our code less and are less able to have coherent big-picture plans concerning the code at-large.

deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
vouaobrasil 1y ago
The ultimate purpose of AI is indeed to remove cognitive autonomy. Little pieces here and there may seem empowering, but added all up, it pretty much takes control away from people.
- nemesisithetic 1y ago
  Bingo. This is a drive to go back to the good old days of leasing mainframes, before the peasants got their grubby hands on the means of computation.

qsort 1y ago

It's extremely hard to measure productivity correctly and self-reports are worthless. I don't think AI tools are a net negative in the average case (people are definitely indexing too much on that goddamn METR article) but "i'm 10x more productive, source trust me bro" is equally nonsense. Using AI tooling means, at least in part, betting on the future.

didericis 1y ago
> Using AI tooling means, at least in part, betting on the future. It means betting on a particular LLM centric vision of the future. I’m still agnostic on that. I think LLMs allow for the creation of a lot of one off scripts and things for people that wouldn’t otherwise be coding, but I have yet to be convinced that more AI usage in a sufficiently senior software development team is more valuable than the traditional way of doing things. I think there’s a fundamental necessity for a human to articulate what a given piece of software should do with a high level of specificity that can’t ever be avoided. The best you can do is piggy back off of higher level language and abstractions that guess what the specifics should be, but I don’t think it’s realistic to think all combinations of all business logic and ui can be boiled down to common patterns that an LLM could infer. And even if that were true, people get bored/like novelty enough that they’ll always want new human created stuff to shove into the training set.
- danielmarkbruce 1y ago
  An LLM is not a tool to allow you to add a layer of abstraction. It's a worker.
  - didericis 1y ago
    It works by translating language abstractions to code. The probability of plain english being correctly translated to code depends on existing code and documented abstractions describing lower level functionality.
    - danielmarkbruce 1y ago

pfisherman 1y ago

> Personally, I’ve been seeing the number of changes for a PR starting to reach into the mid-hundreds now. And fundamentally the developers who make them don’t understand how they work. Could this be fixed by adjusting how tickets are scoped?

chocolatemario 1y ago

This just sounds like a low quality bar or PRs that are too wide in large in scope for AI. In any case, it sounds like taking a single stab at it with an LLM and calling it good. I’m not really AI for everything type of dev, but there are some tasks that AI excels at. If you’re doing it for feature work on a high churn product without a tight grip on the reins, I fear for your products future.

ratelimitsteve 1y ago

>I’ll ask them something about the design and they’ll reply: “IDK Claude did that.” I would want someone entirely off of my team if they did that. Anyone who pushes code they don't understand at least well enough to answer "What does that do?" and "Why did you do it that way?" deserves for their PR to be flat out rejected in whole, not just altered.

rafaelmn 1y ago

I've seen the worst impact on mid/junior level devs. Where they would have to struggle through a problem before - AI is like a magic shortcut to something that looks like it works. And then they submit that crap and I can't trust them again - they will give me AI code without even understanding what it does fully. It robbed them of the learning process and made them even less useful, while making it seem to them they were achieving something. I'm seeing these kinds of people getting removed from workforce fast - you can probably prompt the AI better on your own, have one less layer of indirection and it will be faster.

danielmarkbruce 1y ago
Exactly this. In the right hands the outputs from LLMs are amazing. Not always right, but can be checked. In the wrong hands they are worse than useless because they cannot be checked.

nlawalker 1y ago

> I’ll ask them something about the design and they’ll reply: “IDK Claude did that.” "Then your job is to go ask Claude and get back to me. On that note, if that's what I'm paying you for now, I might be paying you too much..." I'm really interested to see how the intersection of AI and accountability develops over the next few years. It seems like everyone's primary job will effectively be taking accountability for the AI they're driving, and the pay will be a function of how much and what kind of accountability you're taking and the overall stakes.

moritzwarhier 1y ago
I hope so, in reality, people who review code are not always the superiors of the other party :) Not meaning to put on a holier-than-thou hat here. But I hated this many times already. E.g. having to approve code that contains useless comments that were clearly generated by AI, but on the other hand, it's nitpicking to bring it up during review... If my colleagues would do that more often and for more parts of the code, that's where I really start getting inclined to look for a new job.

SatvikBeri 1y ago

A new coworker solved a problem that we'd basically been unable to fix for 2 years, and we were relying on workarounds. This was especially impressive because it was totally outside his expertise. When I asked how he did it, a big part of his process was using LLMs. For the most part, I've noticed my LLM-using coworkers producing better PRs, but I haven't noticed anyone producing more.

subarctic 1y ago
That's the thing, there's a big difference between using AI to help you solve a problem and using it to write code that you don't understand (and not take the time to understand it). It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did. But these are powerful tools and they're not going away anytime soon
- dragonwriter 1y ago
  > It would suck to be reviewing someone else's PR and find out you're putting more effort into understanding their code than they did. It does, but that experience was a real thing long before AI-generated code.

josefritzishere 1y ago

I too have seen anecdotes that people "feel" more productive. But science says that they're 19% less productive. I'm going to go with science on this one.

vouaobrasil 1y ago
Further, the problem with the word "productive" is that it's often measured in quantity and speed, not quality.
davely 1y ago
The study that said devs were 19% less productive was flawed in a number of ways: small sample size, poorly chosen tasks, no prior training on AI tools. The science is by no means conclusive. A joint Harvard / MIT study found developers completed tasks up to 50% faster [0]! [0] https://arxiv.org/abs/2302.06590 https://arxiv.org/abs/2302.06590