21 ms·

My most recent example of this is mentoring young, ambitious, but inexperienced interns. Not only did they produce about the same amount of code in a day that

My most recent example of this is mentoring young, ambitious, but inexperienced interns. Not only did they produce about the same amount of code in a day that they used to produce in a week (or two), several other things made my work harder than before: - During review, they hadn't thought as deeply about their code so my comments seemed to often go over their heads. Instead of a discussion I'd get something like "good catch, I'll fix that" (also reminiscent of an LLM). - The time spent on trivial issues went down a lot, almost zero, the remaining issues were much more subtle and time-consuming to find and describe. - Many bugs were of a new kind (to me), the code would look like it does the right thing but actually not work at all, or just be much more broken than code with that level of "polish" would normally be. This breakdown of pattern-matching compared to "organic" code made the overhead much higher. Spending decades reviewing code and answering Stack Overflow questions often makes it possible to pinpoint not just a bug but how the author got there in the first place and how to help them avoid similar things in the future. - A simple, but bad (inefficient, wrong, illegal, ugly, ...) solution is a nice thing to discuss, but the LLM-assisted junior dev often cooks up something much more complex, which can be bad in many ways at once. The culture of slowly growing a PR from a little bit broken, thinking about design and other considerations, until its high quality and ready for a final review doesn't work the same way. - Instead of fixing the things in the original PR, I'd often get a completely different approach as the response to my first review. Again, often broken in new and subtle ways. This lead to a kind of effort inversion, where senior devs spent much more time on these PRs than the junior authors themselves. The junior dev would feel (I assume) much more productive and competent, but the response to their work would eventually lack most of the usual enthusiasm or encouragement from senior devs. How do people work with these issues? One thing that worked well for me initially was to always require a lot of (passing) tests but eventually these tests would suffer from many of the same problems

agumonkey 1y ago
Somehow interesting how this is similar to other uses of ML driven tools, like electronics engineering where solutions would be near impossible to understand for experienced engineers.
epolanski 1y ago
> the code would look like it does the right thing but actually not work at all, or just be much more broken than code with that level of "polish" would normally be I don't understand, if they don't test the code they write (even if manually) it's not an LLM issue, it's a process one. They have not been taught what does it mean to have a PR ready for being reviewed, LLMs are irrelevant here.
- dakiol 1y ago
  LLMs amplify the problem, so they are not that irrelevant.
- skydhash 1y ago
  Sometimes, orgs don!t mandate testing or descriptive PRs, and then you requiring it makes you look like a PITA.
  - Cthulhu_ 1y ago
    PITA or senior developer that's too senior for that company? Honestly I think an organization has no say in discussions about testing or descriptive PRs, and on the other side, a decent developer does not defer to someone higher-up to decide on the quality of their work.
    - skydhash 1y ago
      Some managers will do anything for velocity, even if the direction is towards a cliff with with sharp rocks below. You try to produce quality work and others are doing tornado programming all over the codebase and be praised for it.
    - mrguyorama 1y ago
      The industry does not care what engineers think because engineers have no power at all. We will not be able to hold any line against a management class that has been assured that AI tooling doubles engineer productivity and have zero ability to judge that claim, because they've never built anything and run into the exact problems that AI tools cause; A lack of understanding. They don't even know what they don't know. If you wanted software engineers to be able to hold any sort of quality line against a few trillion dollars worth of AI investment, we needed to unionize or even form a guild twenty years ago.
stuartjohnson12 1y ago
In the medium term I think you have to shift the work upstream to show that they've put in the labour to actually design the feature or the bug fix. I think we've always had this mental model which needs to change that senior engineers and product managers scope and design features, IC developers (including juniors for simpler work) implement them, and then senior engineers participate in code review. Right now I can't see the value in having a junior engineer on the team who is unable to think about how certain features should be designed. The junior engineer who previously spent his time spinning tires trying to understand the codebase and all the new technologies he has to get to grips with should instead spend that time trying to figure out how that feature fits into the big picture, consider edge cases, and then propose a design for the feature. There are many junior engineers who I wouldn't trust with that kind of work, and honestly I don't think they are employable right now. In the short term, I think you just need to communicate this additional duty of care to make sure that your pull requests are complete because otherwise there's an asymmetry of workload and judge those interns and juniors on how respectful of that they are.
- imiric 1y ago
  I don't think the junior/senior distinction is useful in this case. All software engineers should care about the quality of the end product, regardless of experience. I've seen "senior" engineers doing the bare minimum, and "junior" engineers putting vastly more care into their work. Experience is something that is accrued over time, which gives you more insight into problems you might have seen before, but if there's no care about the product, then it's hardly relevant. The issue with LLM tools is that they don't teach this. The focus is always on getting to the end result as quickly as possible, skipping any of the actually important parts of software development. The way problem solving is approached with LLMs is by feeding them back to the LLM, not by solving them yourself. This is another related issue: relying on an LLM doesn't give you software development experience. That is gained by actually solving problems yourself; understanding how the system works, finding the underlying root cause, fixing it in an elegant way that doesn't create regressions, writing robust tests to ensure it doesn't happen again, etc. This is the learning experience. LLMs can help with this, but they're often not used in this way.
- 1y ago
thisoneisreal 1y ago
I think this is going to look a lot like the same problem in education, where the answer is that we will have to spend less time consuming written artifacts as a form of evaluation. I think effective code reviews will become more continuous and require much more checking in, asking for explanations as the starting point instead of "I read all of your code and give feedback." That just won't be sustainable given the rate at which text can now be output. AI creates the same problem for hiring too: it generates the appearance of knowledge. The problem you and I have as evaluators of that knowledge is there is no other interface to knowledge than language. In a way this is like the oldest philosophy problem in existence. Socrates spent an inordinate amount of time railing against the sophists, people concerned with language and argument rather than truth. We have his same problem, only now on an industrial scale. To your point about tests, I think the answer is to not focus on automated tests at first (though of course you should have those eventually), but instead we should ask people to actually run the code while they explain it to show it working. That's a much better test: show me how it works, and explain it to me.
- skydhash 1y ago
  > instead we should ask people to actually run the code while they explain it to show it working. That's a much better test: show me how it works, and explain it to me. There’s a reason no one does it. Because it’s inefficient. Even in recorded video format. The helpful things are tests and descriptives PRs. The former because its structure is simple enough that you can judge it, and the test run can be part of the commit. The second is for the simple fact that if you can write clearly about your solution, I can the just do a diff of what you told me and what the code is doing, which is way faster than me trying to divine both from the code.
- zeroCalories 1y ago
  I think asking people to explain is good, but it's not scalable. I do this in interviews when I suspect someone is cheating, and it's very easy to see when they've produced something that they don't understand. But it takes a long time to run through the code, and if we had to do that for everything because we can't trust our engineers anymore that would actually decrease productivity, not increase it.
Foreignborn 1y ago
I have a team that’s somewhat junior at a big company. We pretty much have everyone “vibe plan” significantly more than vibe code. - you need to think through the product more, really be sure it’s as clarified as it can be. Everyone has their own process, but it looks like rubber ducking, critiquing, breaking work into phases, those into tasks, etc. (jobs to be done, business requirement docs, domain driven design planning, UX writing product lexicon docs, literally any and all artifacts) - Prioritize setting up tooling and feedback loops (code quality tools of any and every kind, are required). this includes custom rules to help enforce anything you decided during planning. Spent time on this and life will be a lot better for everyone. - We typically making very very detailed plans, and then the agents will “IVI” it (eg automatic linting, single test, test suite, manual evaluation). You basically set up as many and as diverse of automatic feedback signals as you can. —- I will plan and document for 2-4 hours, then print a bunch of small “PRDs” that are like “1 story point” small. There’s clear definitions of done. Doing this, I can pretty much go the gym or have meetings or whatever for 1-2 hours hands off. —-
- lionkor 1y ago
  I pray for whoever has to review code you didn't bother writing
  - Sammi 1y ago
    Software is going to be of two types: 1. Mostly written by LLMs, and only superficially reviewed by humans. 2. Written 50-50% by devs and LLMs. Reviewed to the same degree as now. Software of type 2 will be more expensive and probably of higher quality. Type 1 software will be much much more common, as it will be cheaper. Quality will be lower, but the open question is whether it will be good enough for the use cases of cheap mass produced software. This is the question that is still unanswered by practical experience, and it's the question that all the venture capitalists a salivating about.
    - danaris 1y ago
      I 100% guarantee you there will be plenty of software still written fully by humans—and even more that's written 95% by humans, with minor LLM-based code autocomplete or boilerplate generation.
imiric 1y ago
Well said. That has been my experience as well, but from the perspective of using these tools on my own. Sure, I can now generate thousands of lines of code relatively quickly, but the hard part is actually reviewing the code to ensure that it does what I asked, fix bugs, hunt for security issues, refactor, simplify and remove code, and so on. I've found that it's often much more productive to write the code myself, and rely on the LLM for simple autocomplete tasks on the way. I imagine that this workflow would be much harder when you have to communicate with a less experienced human who will in turn need to translate it to an LLM, because of the additional layers of indirection. I suspect that the majority of the people who claim that these tools are making them more productive are simply skipping these tasks altogether, or they never cared to do them in the first place. Then the burden for maintaining code quality is on the few who actually care, which has now grown much larger because of the amount of code that's thrown at them. Unfortunately, these people are often seen as pedants and sticklers who block PRs for no good reason. That sometimes does happen, but most of the time, these are the folks who actually care about the product shipped to users. I don't have a suggestion for improving this, but rather a grim outlook that it's only going to get worse. The industry will continue to be flooded by software developers trained on LLM use exclusively, and the companies who build these tools will keep promoting the same marketing BS because it builds hype, and by extension, their valuation.
- fhd2 1y ago
  > I suspect that the majority of the people who claim that these tools are making them more productive are simply skipping these tasks altogether I think that's probably true, but I think there are multiple layers here. There's what's commonly called vibe coding, where you don't even look at the code. Then there's what I'd call augmented coding, where you generate a good chunk of the code, but still refactor and generally try to understand it. And then there's understanding every line of it. For this, in particular, I don't believe LLMs speed things up. You can get the LLM to _explain_ every line to you, but what I mean is to look at documentation and specs to build your understanding and test out fine grained changes to confirm it. This is something you naturally do while writing code, and unless you type comically slow, I'm not convinced it's not faster this way around. There's a very tight feedback loop when you are writing and testing code atomically. In my experience, this prevents an unreasonable amount of emergencies and makes debugging orders of magnitude faster. I'd say the bulk of my work is either in the second or the third bucket, depending on whether it's production code, the risks involved etc. These categories have existed before LLMs. Maybe the first two are cheaper now, but I've seen a lot of code bases that fall into them - copy pasting from examples and SO. That is, ultimately, what LLMs speed up. And I think it's OK for some software to fall into these categories. Maybe we'll see too much fall into them for a while. I think eventually, the incredibly long feedback cycles of business decisions will bite and correct this. If our industry really flies off the handle, we tend to have a nice software crisis and sort it out. I'm optimistic that, whatever we land on eventually, generative AI will have reasonable applications in software development. I personally already see some.
dizhn 1y ago
> - Many bugs were of a new kind (to me), the code would look like it does the right thing but actually not work at all, or just be much more broken than code with that level of "polish" would normally be. This reminded me of a quarter million dollar software project one of my employers had contracted to a team in a different country. On the face of it - especially if you go and check by the spec sheet - everything was there but the thing was not a cohesive whole. They did not spend one second beyond the spec sheet and none of the common sense things that "follow" from the spec were there. The whole thing was scrapped immediately. With LLMs this kind of work now basically becomes free to do and automatic.
- AbstractH24 1y ago
  Cheap labor with low EI is/has been what will suffer most from generative AI.
  - vages 1y ago
    What does EI mean in this sentence? Tried looking it up and found no definition that stood out.
    - steveBK123 1y ago
      Emotional intelligence
- Sammi 1y ago
  I'm expecting to see so much more poor quality software being made. We're going to be swimming in an ocean of bad software. Good experienced devs will be able to make better software, but so many inexperienced devs will be regurgitating so much more lousy software at a pace never seen before, it's going to be overwhelming. Or as the original commenter described, they're already being overwhelmed.
  - deleted 1y ago
    [deleted]
  - 1y ago
oytis 1y ago
I guess answering "you obviously didn't write it, please redo" is not an option, because then you are the dinosaur hindering company's march towards the AI future?
- lionkor 1y ago
  You also are never 100% sure if they wrote it
- Cthulhu_ 1y ago
  Honestly, I don't think it matters who wrote it; ultimately it's about the code and the product, not the individual author. That said, a lazy contribution - substandard code or poorly LLM generated - just wastes your time if your feedback is just put into the LLM again. Setting boundaries then is perfectly acceptable, but this isn't unique to LLMs.
  - deleted 1y ago
    [deleted]
  - skeeter2020 1y ago
    I think there is actually a high positive correlation between quality and ownership though, so maybe it does matter who wrote it.
    - rat9988 1y ago
      It still doesn't matter. You don't reject it based on how it was written but the quality.
- rambambram 1y ago
  Haha, good one. You might make this easier by saying you just checked their code with your own AI system and then say it returned "you obviously didn't write it, please redo".
Nextgrid 1y ago
> How do people work with these issues? You give up, approve the trash PRs, wait for it to blow up in production and let the company reap the rewards of their AI-augmented workforce, all while quietly looking for a different job or career altogether.
flir 1y ago
I've found "write it, then ask the chatbot for a code review" to be a good pattern. You have to be judicious about what you accept, but it's often good at tidying things up or catching corner cases I didn't consider. Reading your comment, it occurs to me that a junior could get into a lot of trouble with this pattern.
xiphias2 1y ago
I work alone, not in teams, but use LLM (codex-1) a lot, and it's extremely helpful. I accepted that in return the code base is much lower quality than if I would have written it. What works for me is that after having lots of passing tests, I start refactoring the tests to get closer to property testing: basically prove that the code works by allowing it to go through complex scenarios and check that the state is good in every step instead of just testing lots of independent cases. The better the test is, the harder LLMs are able to cheat.
- steveBK123 1y ago
  I wonder how this trade-off will age. I'm not a Mag7/Saas/SV startup tech guy, so I've tended to work on systems that are in service & maintained for upwards of 10 years. It's not unusual to see 20 year old codebases in my field. We scoff at clever code thats hard to understand leading to poor ability for teams to maintain, but what about knowingly much lower quality code?
  - CharlieDigital 1y ago
    When the price of building becomes low, you just toss it and build more. Much like Ikea's low cost replaceable furniture has replaced artisan, hand made furniture and cheap plastic toys have replaced finely made artifacts. LLM produced code is cheap and low effort; meant to be discarded. In recognizing this, then it should be used where you have this in mind. You might still buy a finely made sofa because it's high touch. But maybe the bookshelves from Ikea are fine.
    - LtWorf 1y ago
      Ikea has a lot of hardwood stuff that will last longer than you and your progeny…
  - LtWorf 1y ago
    In startups there's no real plan to even live that long. Many of them just exist to con investors. So it's not so important.
stoneyhrm1 1y ago
> "good catch, I'll fix that" I see this a lot and even done so myself, I think a lot of people in the industry are a bit too socially-aware and think if they start a discussion they look like they're trying too hard. It's stupid yes, but plenty of times I've started discussions only to be brushed off or not even replied to, and I believed it was because my responses were too long and nobody actually cared.
- jameshart 1y ago
  That doesn’t sound like ‘social awareness’, it sounds like paranoia
- Cthulhu_ 1y ago
  I feel the same way; we use Gitlab in our day to day, and often I find myself writing a long reply after fixing a code review issue, describing what I changed, resources used, etc... then hitting the "resolve" button, which collapses the comment and unless the reviewer has enabled notifications and actually reads them, I doubt they would ever see my well thought-out response. But then, for me, writing is a way to organize thought as well, plus these remarks will stay in the thread for future reference. In theory anyway, in practice it's likely they'll switch from Gitlab to something else and all comments will be lost forever. Which makes me wish for systems that archive review remarks into Git somehow. I'm sure they exist, but they're not commonly used.
aleph_minus_one 1y ago
Simply require from the junior developers that each pull request has to satisfy a very high standard. If they are not sure about something, they may ask, but if they send you some pull request of bad quality to review, and you find something, they deserve a (small) tantrum. It is likely not possible to completely forbid junior developers from using AI tools, but any pull request that they create that contains (AI-generated) code that they don't fully comprehend (they can google) will be rejected (to test this, simply ask them some non-trivial questions about the code). If they do so, again, these junior developers deserve a (small) tantrum.
- imiric 1y ago
  The thing is that a "very high standard" is not a measurable criterion. The project can have test coverage requirements and strict linting to catch basic syntax and logic problems, but how do you enforce simplicity, correctness, robustness, or ergonomics? These are abstract concepts that are difficult to determine, even for experienced developers, so I wouldn't expect less experienced developers to consider them. A code review process is still important, with or without LLMs. So we can ask everyone using these tools to understand the code before submitting a PR, but that's the best we can do. There's no need to call anyone out for not meeting some invisible standard of quality.
  - aleph_minus_one 1y ago
    The answer is simple: "a very high standard" is what the very experienced developers of the team consider to be "a very high standard". :-)
    - mrguyorama 1y ago
      The developers who try to hold a "very high standard" will be accused of blocking progress and fired.
      - aleph_minus_one 1y ago
        Under a bad management, this might indeed (unfortunately) happen. :-(
2OEH8eoCRo0 1y ago
Why isn't your first question, "how did you test this?"
- andrelaszlo 1y ago
  You're right, I am starting to develop that habit.
UncleMeat 1y ago
> Instead of fixing the things in the original PR, I'd often get a completely different approach as the response to my first review. Again, often broken in new and subtle ways. I didn't expect this initially but I am seeing it a ton at work now and it is infuriating. Some big change lands in my lap to review and it has a bunch of issues but they can ultimately be worked out. Then kaboom it is an entirely different change that I need to review from scratch. Usually the second review is just focused on the edits that fixed the comments from my first review. But now we have to start all over.
- swader999 1y ago
  The human PR/code review needs to be abandoned. I'm not sure how or what will replace it. Some kind of programmatic agent review/test loop, contractual code that meets SLAs, vertical slice architecture, microservices (shudder)...
cookiengineer 1y ago
I wanted to add to your points that I think that there's a lack of understanding in architecture, which the previous generation has learned through refactoring and unit tests. If LLMs will be able to write unit tests, this will get worse, because there will be no time spent reflecting about "what do I need" or "how can this be simplified". These are, in my opinion, how to characterize the differences between a Developer, Engineer, and Architect mindset. And LLMs / vibe coding will never develop actual engineers or architects, because they never can develop that mindset. The easiest programming language to spot those architectural mistakes in is coincidentially the one with the least syntax burden. In Go it's pretty easy to discover these types of issues in reviews because you can check the integrated unit tests, which help a lot in narrowing down the complexities of code branches (and whether or not a branch was reached, for example). In my opinion we need better testing/review methodologies. Fuzz testing, unit testing and integration testing isn't enough. We need some kind of logical inference tests which can prove that code branches are kept and called, and allow to confirm satisfiabilities.
a_bonobo 1y ago
>This lead to a kind of effort inversion, where senior devs spent much more time on these PRs than the junior authors themselves. It's funny, I have the same problem, but with subject matter expertise. I work with internal PR people and they clearly have shifted their writing efforts to be AI-assisted or even AI-driven. Now I as the SME get these AI-written blog posts and press releases and I spend a far more time on getting all the hallucinations out of these texts. It's an effort inversion, too - time spent correcting the PR-people's errors has tripled or quadrupled. They're supposed to assist me, not the other way around. I'm not the press release writer here. And of course they don't 'learn' like your junior engineers - it's always AI, it's always different hallucinations. P.S.: And yes I've raised this internally with our leadership - at this rate we'll have 50% of the PR people next year, they're making themselves unemployed. I don't need a middleman who's job it is to copy-paste my email into ChatGPT, then send me the output; I can do that myself.
- dkdbejwi383 1y ago
  Part of the solution is pushing back when you spot tons of obvious lazy LLM errors instead of fixing them yourself. Otherwise there's not much incentive for them to improve their effort.
  - LtWorf 1y ago
    I tried but my boss told me to get used to it. So now I no longer review code at all.
  - a_bonobo 1y ago
    Yes I've tried to have an internal standard for AI usage: at least the PR people have to tell us if they use AI. It completely changes how we approach editing of a text AI-written vs human-written (humans don't hallucinate citations, for a start). Of course this is impossible to enforce, and I believe that the PR people would rather hide their AI usage. (As I wrote above why pay high salaries to people who automate themselves away?)
- kevinventullo 1y ago
  … I can do that myself. So then you see where this is going.
fny 1y ago
Have them first write a "code spec" in the repo with all the interfaces defined and comments that describe the behaviors. """ This is the new adder feature. Internally it uses chained Adders to multiply: Adder(Adder(Adder(x, y), y), ...) """ class Adder: # public attributes x and y def __init__(self, x: float, y: float) -> None: raise NotImplementedError() def add(self) -> float: raise NotImplementedError() class Muliplier: # public attributes x and y # should perform multiplication with repeated adders def __init__(self, x: float, y: float) -> None: raise NotImplementedError() def multiply(self) -> float: raise NotImplementedError() This is a really dumb example (frankly something Claude would write), but it illustrates that they should do this for external interfaces and implementation details. For changes, you'd do the same thing. Specify it as comments and "high level" code ("# remove this class and switch to Multiplier") etc. Then spec -> review -> tests -> review -> code -> review. Depending on how much you trust a dev, you can kill some review steps. 1. It's harder to vibe good specs like this from the start, and prevents Claude from being magical (e.g. executing code to make sure things work) 2. You're embedding a design process into reviews which is useful even if they're coding by hand. 3. It simplifies reviewing generated code because at least the interfaces should be respected. This is the pattern I've been using personally to wrangle ChatGPT and Claude's behavior into submission.
akkad33 1y ago
Could you tell which language they were coding in?
- andrelaszlo 1y ago
  A mix, but a majority Ruby, with some shell scripts and Terraform. My gut feeling is that it would generalize to typed languages, Go, Erlang, even Haskell etc, but maybe some of them make life easier for the reviewer in some ways? What are your thoughts on that?
  - akkad33 1y ago
    I've worked with some junior developers and my experience has been the same as yours. We work primarily with typed languages. With junior developers I see two ways it goes: either they write code that does not take advantage of the type system, typing everything as basic string or numeric types. On the other end of the spectrum, those who go to the extremes where they build a complicated type hierarchy with abstractions more complicated than the problem space requirements
exiguus 1y ago
Thanks for this insides. I am curious and want to know: Is it also a 'good catch, I'll fix that' when you pair program or mob? Or better, did you notice any differences in behavior and issues while pair or mob programming with juniors (instead of using pull requests)?
jedimastert 1y ago
> - During review, they hadn't thought as deeply about their code so my comments seemed to often go over their heads. Instead of a discussion I'd get something like "good catch, I'll fix that" (also reminiscent of an LLM). Would you mind drilling down into this a bit more? I might be dealing with a similar problem and would appreciate if you have any insight
- acedTrex 1y ago
  Basically the juniors just ask the LLM for an explanation of what the problem is and then fix what the LLM interprets your review to be talking about. The way that you solve this is that you pull your junior into a call and work them through your comments one by one verbally, expecting them to comprehend the issues every time.
  - LtWorf 1y ago
    If my boss is ok with AI, why bother and make the junior hate you? The codebase will go to shit regardless.
    - acedTrex 1y ago
      I mean obviously if you have completely checked out and no longer care then don't bother with this.
- andrelaszlo 1y ago
  The "good catch" thing is something I do, too, but mostly for short review comments like "this will blow up if x is null" etc. I had to think a bit about it, but when it feels off it can be something like: - I wrote several paragraphs explaining my reasoning, expecting some follow-up questions. - The "fix" didn't really address my concerns, making it seem like they just said "okay" without really trying to understand. (The times when the whole PR is replaced makes it seem like my review was also just forwarded to the LLM, haha) - I'm also comparing to how I often (especially earlier in my career) thought a lot about how to solve things, and when I got constructive feedback it felt pretty rewarding - and I could often give my own reasoning for why I did things a certain way. Sometimes I had tried a bunch of the things that the reviewer suggested, leading to a more lively back-and-forth. This could just be me, of course, or a cultural thing, but my expectation also comes from how other developers I've worked with react to my reviews. Does that make sense? I'd be interested in hearing more about the problem you're dealing with. If this is not the right place, feel free to send an email :)
exe34 1y ago
> - Instead of fixing the things in the original PR, I'd often get a completely different approach as the response to my first review. Again, often broken in new and subtle ways. This kind of thing drove me mad even before LLMs or coding - it started at school when I helped people with homework. People would insist on switching to an entirely different approach midway through explaining how to fix the first one.
ericyd 1y ago
> always require a lot of (passing) tests My favorite LLM-generated code I've seen in PRs lately is expect(true).toBe(true) Look ma! Tests aren't flaky anymore!
acedTrex 1y ago
The standard competency markers we use to judge code have been hijacked. The new world is very low trust and very painful.
diogolsq 1y ago
Code review has become the new bottleneck, since it’s the layer that prevents sloppy AI-generated code from entering the codebase. One thing I do that helps clean things up before I send a PR is writing a summary. You might consider encouraging your peers to do the same. ## What Changed? Functional Changes: - New service for importing data - New async job for dealing with z. Non-functional Changes: - Refactoring of Class X - Removal of outdated code It might not seem like much, but writing this summary forces you to read through all the changes and reflect. You often catch outdated comments, dead functions left after extractions, or other things that can be improved—before asking a colleague to review it. It also makes the reviewer’s life easier, because even before they look at the code, they already know what to expect.
- imiric 1y ago
  Ha. Almost always when I see PRs with such summaries I can assume that both the summary and the code has been AI-generated. PRs in general shouldn't require elaborate summaries. That's what commit messages are for. If the PR includes many commits where a summary might help, then that might be a sign that there should be multiple PRs.
  - bloomca 1y ago
    Depends on the business logic, sometimes summaries (or a short demo explanation) help a lot to understand the made tradeoffs, so the reviewer can contribute more without spending too much time. It is especially helpful if the part is somewhat isolated. But most of the time it is not very necessary.
  - diogolsq 1y ago
    1. about looking artificial. Granted, it is not only summaries that go into the description—how to test, if there is any pre-deploy or post-deploy setup, any concerns, external documentation, etc. Less is more. A summary serves to clarify, not to endlessly add useless information. ⸻ 2. about the usefulness of summaries. Summaries always provide better information—straight to the point—than commits (which are historical records). This applies to any type of information. When you’re reporting a problem by going through historical facts, it can lead to multiple narratives, added complexity, and convoluted information. Summaries that quickly deliver the key points clearly and focus only on what’s important offer a better way to communicate. If the listener asks for details, they already have a clear idea of what to expect. A good summary is a good introduction to what you are going to see in the commits messages and in the code changes. ______________________ 3.About multiple Prs. Summary helps to clarify what is scope creep (be it a refactor or unrelated code to the ticket); it make it easier for the reviewer demand a split in multiple PRs. examples: A non-summary PR/MR might lead to the question—“WHY is this code here?" "he touched a class here, was he fixing something that the test missed out ? or is just a refactor?" _______________ As a reviewer you can get those information by yourself, although summary helps you to get it much quicker.
nurettin 1y ago
If they insist on using LLMs to generate trash, just use LLMs to do trash reviews on their code.
lubujackson 1y ago
The original approach was to be a surgeon and minimally cut the code to save the patient (the PR). You need to change your thinkong to realize the architecture of the prompt was wrong. Talk in abstractions and let them fully revise the PR, like "this should be refactored to reraise errors to the calling function" instead of pinpointing single lines. In other words, we need to code review the same way we interact with LLMs - point to the overarching flaw and request a reroll.
moi2388 1y ago
This is exactly my experience. Plus documentation is no longer being read because the LLM already generated the code, so the juniors don’t even know what to check before handing in their PR
ksri 1y ago
Struggling with the same issues with junior developers. I've been asking for an implementation plan and iterating on it. Typical workflow is to commit the implementation plan and review it as part of a pr. It takes 2-3 iterations to get right. Then the developer asks claude code to implement the based on the markdown. I've seen good results with this. Another thing I do is ask for the claude session log file. The inputs and thought they provided to claude give me a lot more insight than the output of claude. Quite often I am able to correct the thought process when I know how they are thinking. I've found junior developers treat claude like a sms - small ambiguous messages with very little context, hoping it would perform magic. By reviewing the claude session file, I try to fix this superficial prompting behaviour. And third, I've realized claude works best of the code itself is structured well and has tests, tools to debug and documentation. So I spend more time on tooling so that claude can use these tools to investigate issues, write tests and iterate faster. Still a far way to go, but this seems promising right now.
mixmastamyk 1y ago
These tools seem to work for accelerating seniors, not well for juniors. How are juniors supposed to learn if the aren’t doing?
seanmcdirmid 1y ago
I don’t give my interns green field projects, and they are usually hack jobs like get A working with B, which means they can’t really rely on LLMs to do much of the coding, and must instead must try, run the test, adjust, try again. More like junior investigators who happen to write some code I guess. I imagine this is extremely group-specific though. For junior devs, it’s about the same, I’m assigning hack jobs, because most of what we need to do are hack jobs. The code really isn’t the bottleneck in that case, the research needed to write the code is.
whattho 1y ago
[dead]
singpolyma3 1y ago
Sounds like several fundamental workflow issues that the LLM is perhaps exacerbating but need to be fixed either way. One, they need to run their code. Make sure it works before submitting a PR. If someone submits code to me that does not work I don't care if it came from an LLM or not, go run your code and come back when it works. If they routinely refuse to run their code and never learn their lesson then I might suggest they find another profession... Or require they submit a video of the code working. Second, going away and coming back with a totally different PR I give the feedback of "what happened to the code we were working on before? We didn't need all new code." As the senior my time is worth (a bit) more than the intern's so I don't hesitate to make their bad choices their problem. Come back when you've made a serious attempt and then we can discuss it.
vjvjvjvjghv 1y ago
That's how it often works with offshore code. You get a huge pile of code that meets the spec, so not totally wrong, but with a lot of small issues that are hard to identify. And you as the senior dev are now in a bad situation: since project management has marked the task as "Done" already, you are the bad guy if you reject the code and ask for rework. At some point you are worn down by all the pressure and let the code through and you end up with a growing pile of questionable code that sort of works but requires a ton of maintenance and is hard to change. You can't win. My only hope is that AI one day will be much better than humans in every aspect and produce super high quality code. I don't see why this wouldn't happen. The current tools are still primitive.
throwaway2037 1y ago
I see these posts on HN from time to time. Do you really think your code was any better when you were an intern? Mine was god awful. They probably binned all my work after each internship! I don't think an LLM makes them any worse. Here is the crazy part: As a nearly neckbeard, there were no code reviews or PRs in my era. And mostly zero unit tests.
jbverschoor 1y ago
They’re not juniors. They’re freshmen
martin82 1y ago
these are excellent and well put observations! thank you for taking the time to share this.

agumonkey 1y ago

Somehow interesting how this is similar to other uses of ML driven tools, like electronics engineering where solutions would be near impossible to understand for experienced engineers.

epolanski 1y ago

> the code would look like it does the right thing but actually not work at all, or just be much more broken than code with that level of "polish" would normally be I don't understand, if they don't test the code they write (even if manually) it's not an LLM issue, it's a process one. They have not been taught what does it mean to have a PR ready for being reviewed, LLMs are irrelevant here.

dakiol 1y ago
LLMs amplify the problem, so they are not that irrelevant.
skydhash 1y ago
Sometimes, orgs don!t mandate testing or descriptive PRs, and then you requiring it makes you look like a PITA.
- Cthulhu_ 1y ago
  PITA or senior developer that's too senior for that company? Honestly I think an organization has no say in discussions about testing or descriptive PRs, and on the other side, a decent developer does not defer to someone higher-up to decide on the quality of their work.
  - skydhash 1y ago
    Some managers will do anything for velocity, even if the direction is towards a cliff with with sharp rocks below. You try to produce quality work and others are doing tornado programming all over the codebase and be praised for it.
  - mrguyorama 1y ago
    The industry does not care what engineers think because engineers have no power at all. We will not be able to hold any line against a management class that has been assured that AI tooling doubles engineer productivity and have zero ability to judge that claim, because they've never built anything and run into the exact problems that AI tools cause; A lack of understanding. They don't even know what they don't know. If you wanted software engineers to be able to hold any sort of quality line against a few trillion dollars worth of AI investment, we needed to unionize or even form a guild twenty years ago.

stuartjohnson12 1y ago

In the medium term I think you have to shift the work upstream to show that they've put in the labour to actually design the feature or the bug fix. I think we've always had this mental model which needs to change that senior engineers and product managers scope and design features, IC developers (including juniors for simpler work) implement them, and then senior engineers participate in code review. Right now I can't see the value in having a junior engineer on the team who is unable to think about how certain features should be designed. The junior engineer who previously spent his time spinning tires trying to understand the codebase and all the new technologies he has to get to grips with should instead spend that time trying to figure out how that feature fits into the big picture, consider edge cases, and then propose a design for the feature. There are many junior engineers who I wouldn't trust with that kind of work, and honestly I don't think they are employable right now. In the short term, I think you just need to communicate this additional duty of care to make sure that your pull requests are complete because otherwise there's an asymmetry of workload and judge those interns and juniors on how respectful of that they are.

imiric 1y ago
I don't think the junior/senior distinction is useful in this case. All software engineers should care about the quality of the end product, regardless of experience. I've seen "senior" engineers doing the bare minimum, and "junior" engineers putting vastly more care into their work. Experience is something that is accrued over time, which gives you more insight into problems you might have seen before, but if there's no care about the product, then it's hardly relevant. The issue with LLM tools is that they don't teach this. The focus is always on getting to the end result as quickly as possible, skipping any of the actually important parts of software development. The way problem solving is approached with LLMs is by feeding them back to the LLM, not by solving them yourself. This is another related issue: relying on an LLM doesn't give you software development experience. That is gained by actually solving problems yourself; understanding how the system works, finding the underlying root cause, fixing it in an elegant way that doesn't create regressions, writing robust tests to ensure it doesn't happen again, etc. This is the learning experience. LLMs can help with this, but they're often not used in this way.
1y ago

thisoneisreal 1y ago

I think this is going to look a lot like the same problem in education, where the answer is that we will have to spend less time consuming written artifacts as a form of evaluation. I think effective code reviews will become more continuous and require much more checking in, asking for explanations as the starting point instead of "I read all of your code and give feedback." That just won't be sustainable given the rate at which text can now be output. AI creates the same problem for hiring too: it generates the appearance of knowledge. The problem you and I have as evaluators of that knowledge is there is no other interface to knowledge than language. In a way this is like the oldest philosophy problem in existence. Socrates spent an inordinate amount of time railing against the sophists, people concerned with language and argument rather than truth. We have his same problem, only now on an industrial scale. To your point about tests, I think the answer is to not focus on automated tests at first (though of course you should have those eventually), but instead we should ask people to actually run the code while they explain it to show it working. That's a much better test: show me how it works, and explain it to me.

skydhash 1y ago
> instead we should ask people to actually run the code while they explain it to show it working. That's a much better test: show me how it works, and explain it to me. There’s a reason no one does it. Because it’s inefficient. Even in recorded video format. The helpful things are tests and descriptives PRs. The former because its structure is simple enough that you can judge it, and the test run can be part of the commit. The second is for the simple fact that if you can write clearly about your solution, I can the just do a diff of what you told me and what the code is doing, which is way faster than me trying to divine both from the code.
zeroCalories 1y ago
I think asking people to explain is good, but it's not scalable. I do this in interviews when I suspect someone is cheating, and it's very easy to see when they've produced something that they don't understand. But it takes a long time to run through the code, and if we had to do that for everything because we can't trust our engineers anymore that would actually decrease productivity, not increase it.

Foreignborn 1y ago

I have a team that’s somewhat junior at a big company. We pretty much have everyone “vibe plan” significantly more than vibe code. - you need to think through the product more, really be sure it’s as clarified as it can be. Everyone has their own process, but it looks like rubber ducking, critiquing, breaking work into phases, those into tasks, etc. (jobs to be done, business requirement docs, domain driven design planning, UX writing product lexicon docs, literally any and all artifacts) - Prioritize setting up tooling and feedback loops (code quality tools of any and every kind, are required). this includes custom rules to help enforce anything you decided during planning. Spent time on this and life will be a lot better for everyone. - We typically making very very detailed plans, and then the agents will “IVI” it (eg automatic linting, single test, test suite, manual evaluation). You basically set up as many and as diverse of automatic feedback signals as you can. —- I will plan and document for 2-4 hours, then print a bunch of small “PRDs” that are like “1 story point” small. There’s clear definitions of done. Doing this, I can pretty much go the gym or have meetings or whatever for 1-2 hours hands off. —-

lionkor 1y ago
I pray for whoever has to review code you didn't bother writing
- Sammi 1y ago
  Software is going to be of two types: 1. Mostly written by LLMs, and only superficially reviewed by humans. 2. Written 50-50% by devs and LLMs. Reviewed to the same degree as now. Software of type 2 will be more expensive and probably of higher quality. Type 1 software will be much much more common, as it will be cheaper. Quality will be lower, but the open question is whether it will be good enough for the use cases of cheap mass produced software. This is the question that is still unanswered by practical experience, and it's the question that all the venture capitalists a salivating about.
  - danaris 1y ago
    I 100% guarantee you there will be plenty of software still written fully by humans—and even more that's written 95% by humans, with minor LLM-based code autocomplete or boilerplate generation.

imiric 1y ago

Well said. That has been my experience as well, but from the perspective of using these tools on my own. Sure, I can now generate thousands of lines of code relatively quickly, but the hard part is actually reviewing the code to ensure that it does what I asked, fix bugs, hunt for security issues, refactor, simplify and remove code, and so on. I've found that it's often much more productive to write the code myself, and rely on the LLM for simple autocomplete tasks on the way. I imagine that this workflow would be much harder when you have to communicate with a less experienced human who will in turn need to translate it to an LLM, because of the additional layers of indirection. I suspect that the majority of the people who claim that these tools are making them more productive are simply skipping these tasks altogether, or they never cared to do them in the first place. Then the burden for maintaining code quality is on the few who actually care, which has now grown much larger because of the amount of code that's thrown at them. Unfortunately, these people are often seen as pedants and sticklers who block PRs for no good reason. That sometimes does happen, but most of the time, these are the folks who actually care about the product shipped to users. I don't have a suggestion for improving this, but rather a grim outlook that it's only going to get worse. The industry will continue to be flooded by software developers trained on LLM use exclusively, and the companies who build these tools will keep promoting the same marketing BS because it builds hype, and by extension, their valuation.

fhd2 1y ago
> I suspect that the majority of the people who claim that these tools are making them more productive are simply skipping these tasks altogether I think that's probably true, but I think there are multiple layers here. There's what's commonly called vibe coding, where you don't even look at the code. Then there's what I'd call augmented coding, where you generate a good chunk of the code, but still refactor and generally try to understand it. And then there's understanding every line of it. For this, in particular, I don't believe LLMs speed things up. You can get the LLM to _explain_ every line to you, but what I mean is to look at documentation and specs to build your understanding and test out fine grained changes to confirm it. This is something you naturally do while writing code, and unless you type comically slow, I'm not convinced it's not faster this way around. There's a very tight feedback loop when you are writing and testing code atomically. In my experience, this prevents an unreasonable amount of emergencies and makes debugging orders of magnitude faster. I'd say the bulk of my work is either in the second or the third bucket, depending on whether it's production code, the risks involved etc. These categories have existed before LLMs. Maybe the first two are cheaper now, but I've seen a lot of code bases that fall into them - copy pasting from examples and SO. That is, ultimately, what LLMs speed up. And I think it's OK for some software to fall into these categories. Maybe we'll see too much fall into them for a while. I think eventually, the incredibly long feedback cycles of business decisions will bite and correct this. If our industry really flies off the handle, we tend to have a nice software crisis and sort it out. I'm optimistic that, whatever we land on eventually, generative AI will have reasonable applications in software development. I personally already see some.

dizhn 1y ago

> - Many bugs were of a new kind (to me), the code would look like it does the right thing but actually not work at all, or just be much more broken than code with that level of "polish" would normally be. This reminded me of a quarter million dollar software project one of my employers had contracted to a team in a different country. On the face of it - especially if you go and check by the spec sheet - everything was there but the thing was not a cohesive whole. They did not spend one second beyond the spec sheet and none of the common sense things that "follow" from the spec were there. The whole thing was scrapped immediately. With LLMs this kind of work now basically becomes free to do and automatic.

AbstractH24 1y ago
Cheap labor with low EI is/has been what will suffer most from generative AI.
- vages 1y ago
  What does EI mean in this sentence? Tried looking it up and found no definition that stood out.
  - steveBK123 1y ago
    Emotional intelligence
Sammi 1y ago
I'm expecting to see so much more poor quality software being made. We're going to be swimming in an ocean of bad software. Good experienced devs will be able to make better software, but so many inexperienced devs will be regurgitating so much more lousy software at a pace never seen before, it's going to be overwhelming. Or as the original commenter described, they're already being overwhelmed.
- deleted 1y ago
  [deleted]
- 1y ago

oytis 1y ago

I guess answering "you obviously didn't write it, please redo" is not an option, because then you are the dinosaur hindering company's march towards the AI future?

lionkor 1y ago
You also are never 100% sure if they wrote it
Cthulhu_ 1y ago
Honestly, I don't think it matters who wrote it; ultimately it's about the code and the product, not the individual author. That said, a lazy contribution - substandard code or poorly LLM generated - just wastes your time if your feedback is just put into the LLM again. Setting boundaries then is perfectly acceptable, but this isn't unique to LLMs.
- deleted 1y ago
  [deleted]
- skeeter2020 1y ago
  I think there is actually a high positive correlation between quality and ownership though, so maybe it does matter who wrote it.
  - rat9988 1y ago
    It still doesn't matter. You don't reject it based on how it was written but the quality.
rambambram 1y ago
Haha, good one. You might make this easier by saying you just checked their code with your own AI system and then say it returned "you obviously didn't write it, please redo".

Nextgrid 1y ago

> How do people work with these issues? You give up, approve the trash PRs, wait for it to blow up in production and let the company reap the rewards of their AI-augmented workforce, all while quietly looking for a different job or career altogether.

flir 1y ago

I've found "write it, then ask the chatbot for a code review" to be a good pattern. You have to be judicious about what you accept, but it's often good at tidying things up or catching corner cases I didn't consider. Reading your comment, it occurs to me that a junior could get into a lot of trouble with this pattern.

xiphias2 1y ago

I work alone, not in teams, but use LLM (codex-1) a lot, and it's extremely helpful. I accepted that in return the code base is much lower quality than if I would have written it. What works for me is that after having lots of passing tests, I start refactoring the tests to get closer to property testing: basically prove that the code works by allowing it to go through complex scenarios and check that the state is good in every step instead of just testing lots of independent cases. The better the test is, the harder LLMs are able to cheat.

steveBK123 1y ago
I wonder how this trade-off will age. I'm not a Mag7/Saas/SV startup tech guy, so I've tended to work on systems that are in service & maintained for upwards of 10 years. It's not unusual to see 20 year old codebases in my field. We scoff at clever code thats hard to understand leading to poor ability for teams to maintain, but what about knowingly much lower quality code?
- CharlieDigital 1y ago
  When the price of building becomes low, you just toss it and build more. Much like Ikea's low cost replaceable furniture has replaced artisan, hand made furniture and cheap plastic toys have replaced finely made artifacts. LLM produced code is cheap and low effort; meant to be discarded. In recognizing this, then it should be used where you have this in mind. You might still buy a finely made sofa because it's high touch. But maybe the bookshelves from Ikea are fine.
  - LtWorf 1y ago
    Ikea has a lot of hardwood stuff that will last longer than you and your progeny…
- LtWorf 1y ago
  In startups there's no real plan to even live that long. Many of them just exist to con investors. So it's not so important.

stoneyhrm1 1y ago

> "good catch, I'll fix that" I see this a lot and even done so myself, I think a lot of people in the industry are a bit too socially-aware and think if they start a discussion they look like they're trying too hard. It's stupid yes, but plenty of times I've started discussions only to be brushed off or not even replied to, and I believed it was because my responses were too long and nobody actually cared.

jameshart 1y ago
That doesn’t sound like ‘social awareness’, it sounds like paranoia
Cthulhu_ 1y ago
I feel the same way; we use Gitlab in our day to day, and often I find myself writing a long reply after fixing a code review issue, describing what I changed, resources used, etc... then hitting the "resolve" button, which collapses the comment and unless the reviewer has enabled notifications and actually reads them, I doubt they would ever see my well thought-out response. But then, for me, writing is a way to organize thought as well, plus these remarks will stay in the thread for future reference. In theory anyway, in practice it's likely they'll switch from Gitlab to something else and all comments will be lost forever. Which makes me wish for systems that archive review remarks into Git somehow. I'm sure they exist, but they're not commonly used.

aleph_minus_one 1y ago

Simply require from the junior developers that each pull request has to satisfy a very high standard. If they are not sure about something, they may ask, but if they send you some pull request of bad quality to review, and you find something, they deserve a (small) tantrum. It is likely not possible to completely forbid junior developers from using AI tools, but any pull request that they create that contains (AI-generated) code that they don't fully comprehend (they can google) will be rejected (to test this, simply ask them some non-trivial questions about the code). If they do so, again, these junior developers deserve a (small) tantrum.

imiric 1y ago
The thing is that a "very high standard" is not a measurable criterion. The project can have test coverage requirements and strict linting to catch basic syntax and logic problems, but how do you enforce simplicity, correctness, robustness, or ergonomics? These are abstract concepts that are difficult to determine, even for experienced developers, so I wouldn't expect less experienced developers to consider them. A code review process is still important, with or without LLMs. So we can ask everyone using these tools to understand the code before submitting a PR, but that's the best we can do. There's no need to call anyone out for not meeting some invisible standard of quality.
- aleph_minus_one 1y ago
  The answer is simple: "a very high standard" is what the very experienced developers of the team consider to be "a very high standard". :-)
  - mrguyorama 1y ago
    The developers who try to hold a "very high standard" will be accused of blocking progress and fired.
    - aleph_minus_one 1y ago
      Under a bad management, this might indeed (unfortunately) happen. :-(

2OEH8eoCRo0 1y ago

Why isn't your first question, "how did you test this?"

andrelaszlo 1y ago
You're right, I am starting to develop that habit.

UncleMeat 1y ago

> Instead of fixing the things in the original PR, I'd often get a completely different approach as the response to my first review. Again, often broken in new and subtle ways. I didn't expect this initially but I am seeing it a ton at work now and it is infuriating. Some big change lands in my lap to review and it has a bunch of issues but they can ultimately be worked out. Then kaboom it is an entirely different change that I need to review from scratch. Usually the second review is just focused on the edits that fixed the comments from my first review. But now we have to start all over.

swader999 1y ago
The human PR/code review needs to be abandoned. I'm not sure how or what will replace it. Some kind of programmatic agent review/test loop, contractual code that meets SLAs, vertical slice architecture, microservices (shudder)...

cookiengineer 1y ago

I wanted to add to your points that I think that there's a lack of understanding in architecture, which the previous generation has learned through refactoring and unit tests. If LLMs will be able to write unit tests, this will get worse, because there will be no time spent reflecting about "what do I need" or "how can this be simplified". These are, in my opinion, how to characterize the differences between a Developer, Engineer, and Architect mindset. And LLMs / vibe coding will never develop actual engineers or architects, because they never can develop that mindset. The easiest programming language to spot those architectural mistakes in is coincidentially the one with the least syntax burden. In Go it's pretty easy to discover these types of issues in reviews because you can check the integrated unit tests, which help a lot in narrowing down the complexities of code branches (and whether or not a branch was reached, for example). In my opinion we need better testing/review methodologies. Fuzz testing, unit testing and integration testing isn't enough. We need some kind of logical inference tests which can prove that code branches are kept and called, and allow to confirm satisfiabilities.

a_bonobo 1y ago

>This lead to a kind of effort inversion, where senior devs spent much more time on these PRs than the junior authors themselves. It's funny, I have the same problem, but with subject matter expertise. I work with internal PR people and they clearly have shifted their writing efforts to be AI-assisted or even AI-driven. Now I as the SME get these AI-written blog posts and press releases and I spend a far more time on getting all the hallucinations out of these texts. It's an effort inversion, too - time spent correcting the PR-people's errors has tripled or quadrupled. They're supposed to assist me, not the other way around. I'm not the press release writer here. And of course they don't 'learn' like your junior engineers - it's always AI, it's always different hallucinations. P.S.: And yes I've raised this internally with our leadership - at this rate we'll have 50% of the PR people next year, they're making themselves unemployed. I don't need a middleman who's job it is to copy-paste my email into ChatGPT, then send me the output; I can do that myself.

dkdbejwi383 1y ago
Part of the solution is pushing back when you spot tons of obvious lazy LLM errors instead of fixing them yourself. Otherwise there's not much incentive for them to improve their effort.
- LtWorf 1y ago
  I tried but my boss told me to get used to it. So now I no longer review code at all.
- a_bonobo 1y ago
  Yes I've tried to have an internal standard for AI usage: at least the PR people have to tell us if they use AI. It completely changes how we approach editing of a text AI-written vs human-written (humans don't hallucinate citations, for a start). Of course this is impossible to enforce, and I believe that the PR people would rather hide their AI usage. (As I wrote above why pay high salaries to people who automate themselves away?)
kevinventullo 1y ago
… I can do that myself. So then you see where this is going.

fny 1y ago

Have them first write a "code spec" in the repo with all the interfaces defined and comments that describe the behaviors. """ This is the new adder feature. Internally it uses chained Adders to multiply: Adder(Adder(Adder(x, y), y), ...) """ class Adder: # public attributes x and y def __init__(self, x: float, y: float) -> None: raise NotImplementedError() def add(self) -> float: raise NotImplementedError() class Muliplier: # public attributes x and y # should perform multiplication with repeated adders def __init__(self, x: float, y: float) -> None: raise NotImplementedError() def multiply(self) -> float: raise NotImplementedError() This is a really dumb example (frankly something Claude would write), but it illustrates that they should do this for external interfaces and implementation details. For changes, you'd do the same thing. Specify it as comments and "high level" code ("# remove this class and switch to Multiplier") etc. Then spec -> review -> tests -> review -> code -> review. Depending on how much you trust a dev, you can kill some review steps. 1. It's harder to vibe good specs like this from the start, and prevents Claude from being magical (e.g. executing code to make sure things work) 2. You're embedding a design process into reviews which is useful even if they're coding by hand. 3. It simplifies reviewing generated code because at least the interfaces should be respected. This is the pattern I've been using personally to wrangle ChatGPT and Claude's behavior into submission.

akkad33 1y ago

Could you tell which language they were coding in?

andrelaszlo 1y ago
A mix, but a majority Ruby, with some shell scripts and Terraform. My gut feeling is that it would generalize to typed languages, Go, Erlang, even Haskell etc, but maybe some of them make life easier for the reviewer in some ways? What are your thoughts on that?
- akkad33 1y ago
  I've worked with some junior developers and my experience has been the same as yours. We work primarily with typed languages. With junior developers I see two ways it goes: either they write code that does not take advantage of the type system, typing everything as basic string or numeric types. On the other end of the spectrum, those who go to the extremes where they build a complicated type hierarchy with abstractions more complicated than the problem space requirements

exiguus 1y ago

Thanks for this insides. I am curious and want to know: Is it also a 'good catch, I'll fix that' when you pair program or mob? Or better, did you notice any differences in behavior and issues while pair or mob programming with juniors (instead of using pull requests)?

jedimastert 1y ago

> - During review, they hadn't thought as deeply about their code so my comments seemed to often go over their heads. Instead of a discussion I'd get something like "good catch, I'll fix that" (also reminiscent of an LLM). Would you mind drilling down into this a bit more? I might be dealing with a similar problem and would appreciate if you have any insight

acedTrex 1y ago
Basically the juniors just ask the LLM for an explanation of what the problem is and then fix what the LLM interprets your review to be talking about. The way that you solve this is that you pull your junior into a call and work them through your comments one by one verbally, expecting them to comprehend the issues every time.
- LtWorf 1y ago
  If my boss is ok with AI, why bother and make the junior hate you? The codebase will go to shit regardless.
  - acedTrex 1y ago
    I mean obviously if you have completely checked out and no longer care then don't bother with this.
andrelaszlo 1y ago
The "good catch" thing is something I do, too, but mostly for short review comments like "this will blow up if x is null" etc. I had to think a bit about it, but when it feels off it can be something like: - I wrote several paragraphs explaining my reasoning, expecting some follow-up questions. - The "fix" didn't really address my concerns, making it seem like they just said "okay" without really trying to understand. (The times when the whole PR is replaced makes it seem like my review was also just forwarded to the LLM, haha) - I'm also comparing to how I often (especially earlier in my career) thought a lot about how to solve things, and when I got constructive feedback it felt pretty rewarding - and I could often give my own reasoning for why I did things a certain way. Sometimes I had tried a bunch of the things that the reviewer suggested, leading to a more lively back-and-forth. This could just be me, of course, or a cultural thing, but my expectation also comes from how other developers I've worked with react to my reviews. Does that make sense? I'd be interested in hearing more about the problem you're dealing with. If this is not the right place, feel free to send an email :)

exe34 1y ago

> - Instead of fixing the things in the original PR, I'd often get a completely different approach as the response to my first review. Again, often broken in new and subtle ways. This kind of thing drove me mad even before LLMs or coding - it started at school when I helped people with homework. People would insist on switching to an entirely different approach midway through explaining how to fix the first one.

ericyd 1y ago

> always require a lot of (passing) tests My favorite LLM-generated code I've seen in PRs lately is expect(true).toBe(true) Look ma! Tests aren't flaky anymore!

acedTrex 1y ago

The standard competency markers we use to judge code have been hijacked. The new world is very low trust and very painful.

diogolsq 1y ago

Code review has become the new bottleneck, since it’s the layer that prevents sloppy AI-generated code from entering the codebase. One thing I do that helps clean things up before I send a PR is writing a summary. You might consider encouraging your peers to do the same. ## What Changed? Functional Changes: - New service for importing data - New async job for dealing with z. Non-functional Changes: - Refactoring of Class X - Removal of outdated code It might not seem like much, but writing this summary forces you to read through all the changes and reflect. You often catch outdated comments, dead functions left after extractions, or other things that can be improved—before asking a colleague to review it. It also makes the reviewer’s life easier, because even before they look at the code, they already know what to expect.

imiric 1y ago
Ha. Almost always when I see PRs with such summaries I can assume that both the summary and the code has been AI-generated. PRs in general shouldn't require elaborate summaries. That's what commit messages are for. If the PR includes many commits where a summary might help, then that might be a sign that there should be multiple PRs.
- bloomca 1y ago
  Depends on the business logic, sometimes summaries (or a short demo explanation) help a lot to understand the made tradeoffs, so the reviewer can contribute more without spending too much time. It is especially helpful if the part is somewhat isolated. But most of the time it is not very necessary.
- diogolsq 1y ago
  1. about looking artificial. Granted, it is not only summaries that go into the description—how to test, if there is any pre-deploy or post-deploy setup, any concerns, external documentation, etc. Less is more. A summary serves to clarify, not to endlessly add useless information. ⸻ 2. about the usefulness of summaries. Summaries always provide better information—straight to the point—than commits (which are historical records). This applies to any type of information. When you’re reporting a problem by going through historical facts, it can lead to multiple narratives, added complexity, and convoluted information. Summaries that quickly deliver the key points clearly and focus only on what’s important offer a better way to communicate. If the listener asks for details, they already have a clear idea of what to expect. A good summary is a good introduction to what you are going to see in the commits messages and in the code changes. ______________________ 3.About multiple Prs. Summary helps to clarify what is scope creep (be it a refactor or unrelated code to the ticket); it make it easier for the reviewer demand a split in multiple PRs. examples: A non-summary PR/MR might lead to the question—“WHY is this code here?" "he touched a class here, was he fixing something that the test missed out ? or is just a refactor?" _______________ As a reviewer you can get those information by yourself, although summary helps you to get it much quicker.

nurettin 1y ago

If they insist on using LLMs to generate trash, just use LLMs to do trash reviews on their code.

lubujackson 1y ago

The original approach was to be a surgeon and minimally cut the code to save the patient (the PR). You need to change your thinkong to realize the architecture of the prompt was wrong. Talk in abstractions and let them fully revise the PR, like "this should be refactored to reraise errors to the calling function" instead of pinpointing single lines. In other words, we need to code review the same way we interact with LLMs - point to the overarching flaw and request a reroll.

moi2388 1y ago

This is exactly my experience. Plus documentation is no longer being read because the LLM already generated the code, so the juniors don’t even know what to check before handing in their PR

ksri 1y ago

Struggling with the same issues with junior developers. I've been asking for an implementation plan and iterating on it. Typical workflow is to commit the implementation plan and review it as part of a pr. It takes 2-3 iterations to get right. Then the developer asks claude code to implement the based on the markdown. I've seen good results with this. Another thing I do is ask for the claude session log file. The inputs and thought they provided to claude give me a lot more insight than the output of claude. Quite often I am able to correct the thought process when I know how they are thinking. I've found junior developers treat claude like a sms - small ambiguous messages with very little context, hoping it would perform magic. By reviewing the claude session file, I try to fix this superficial prompting behaviour. And third, I've realized claude works best of the code itself is structured well and has tests, tools to debug and documentation. So I spend more time on tooling so that claude can use these tools to investigate issues, write tests and iterate faster. Still a far way to go, but this seems promising right now.

mixmastamyk 1y ago

These tools seem to work for accelerating seniors, not well for juniors. How are juniors supposed to learn if the aren’t doing?

seanmcdirmid 1y ago

I don’t give my interns green field projects, and they are usually hack jobs like get A working with B, which means they can’t really rely on LLMs to do much of the coding, and must instead must try, run the test, adjust, try again. More like junior investigators who happen to write some code I guess. I imagine this is extremely group-specific though. For junior devs, it’s about the same, I’m assigning hack jobs, because most of what we need to do are hack jobs. The code really isn’t the bottleneck in that case, the research needed to write the code is.

whattho 1y ago

[dead]

singpolyma3 1y ago

Sounds like several fundamental workflow issues that the LLM is perhaps exacerbating but need to be fixed either way. One, they need to run their code. Make sure it works before submitting a PR. If someone submits code to me that does not work I don't care if it came from an LLM or not, go run your code and come back when it works. If they routinely refuse to run their code and never learn their lesson then I might suggest they find another profession... Or require they submit a video of the code working. Second, going away and coming back with a totally different PR I give the feedback of "what happened to the code we were working on before? We didn't need all new code." As the senior my time is worth (a bit) more than the intern's so I don't hesitate to make their bad choices their problem. Come back when you've made a serious attempt and then we can discuss it.

vjvjvjvjghv 1y ago

That's how it often works with offshore code. You get a huge pile of code that meets the spec, so not totally wrong, but with a lot of small issues that are hard to identify. And you as the senior dev are now in a bad situation: since project management has marked the task as "Done" already, you are the bad guy if you reject the code and ask for rework. At some point you are worn down by all the pressure and let the code through and you end up with a growing pile of questionable code that sort of works but requires a ton of maintenance and is hard to change. You can't win. My only hope is that AI one day will be much better than humans in every aspect and produce super high quality code. I don't see why this wouldn't happen. The current tools are still primitive.

throwaway2037 1y ago

I see these posts on HN from time to time. Do you really think your code was any better when you were an intern? Mine was god awful. They probably binned all my work after each internship! I don't think an LLM makes them any worse. Here is the crazy part: As a nearly neckbeard, there were no code reviews or PRs in my era. And mostly zero unit tests.

jbverschoor 1y ago

They’re not juniors. They’re freshmen

martin82 1y ago

these are excellent and well put observations! thank you for taking the time to share this.