9 ms·

The best code I've written with an LLM has been where I architect it, I guide the LLM through the scaffolding and initial proofs of different components, and th

by bentt 1y ago

The best code I've written with an LLM has been where I architect it, I guide the LLM through the scaffolding and initial proofs of different components, and then I guide it through adding features. Along the way it makes mistakes and I guide it through fixing them. Then when it is slow, I profile and guide it through optimizations. So in the end, it's code that I know very, very well. I could have written it but it would have taken me about 3x longer when all is said and done. Maybe longer. There are usually parts that have difficult functions but the inputs and outputs of those functions are testable so it doesn't matter so much that you know every detail of the implementation, as long as it is validated. This is just not junior stuff.

mattmanser 1y ago
Would it have actually taken you 3x longer? I am surprising myself these days with how fast I'm being using AI as a glorified Stack Overflow. We are also having studies and posts come out that when actually tried side-by-side, the AI writes the coding route is slower, though the developer percieves it as faster.
- WesolyKubeczek 1y ago
  > the AI writes the coding route is slower, though the developer percieves it as faster. I have this pattern while driving. Using the main roads, when there is little to no traffic, the commute is objectively, measurably the fastest. However, during peak hours, I find myself in traffic jams, so I divert to squiggly country roads which are both slower and longer, but at least I’m moving all the time. The thing is, when I did have to take the main road during the peak traffic, the difference between it and squiggly country roads was like two to three minutes at worst, and not half an hour like I was afraid it would be. Sure, ten minutes crawling or standing felt like an hour. Maybe coding with LLMs makes you think you are doing something productive the whole time, but the actual output is little different from the old way? But hey, at least it’s not like you’re twiddling your thumbs for hours, and the bossware measuring your productivity by your keyboard and mouse activity is happy!
  - realisticfoo 1y ago
    the best use of LLM is to help you program a digispark USB dongle with Arduino to make it emulate mouse movement and scrolling to appease the idi0tic spyware retarded boomers monitor like it was relevant somehow on their corporate spy box. Meanwhile we adults can do real work on a separate real computer. Never use their laptop more than absolutely minimum possible.
    - WesolyKubeczek 1y ago
      Well, you might have several obstacles here: 1) the bossware might take screenshots too 2) the bosses pay for the whole LLM so they expect you to use the whole LLM 3) you may not want to contaminate your spare computer with whatever shit you're working on on job, and indeed it may be considered a breach of security (as if feeding OpenAI/Anthropic isn't, lol, but that's beside the point. So you continue to feel miserable, but you get your paycheck, and it's better than unemployment, and your kids are fed and clothed, so there's that.
- notarobot123 1y ago
  I am not the biggest fan of LLMs but I have to admit that, as long as you understand what the technology is and how it works, it is a very powerful tool. I think the mixed reports on utility have a lot to do with the very different ways the tool is used and how much 'magic' the end-user expects versus how much the end-user expects to guide the tool to do the work. To get the best out of it, you do have to provide significant amount of scaffolding (though it can help with that too). If you're just pointing it at a codebase and expecting it to figure it out, you're going to have mixed results at best. If you guide it well, it can save a significant amount of manual effort and time.
  - bentt 1y ago
    yeah very few months I try to have it “just do magic” again and I re-learn the lesson. Like, I’ll just say “optimize this shader!” and plug it in blind. It doesn’t work. The only way it could is if the LLM has a testing loop itself. I guess in web world it could, but in my world of game dev, not so much. So I stick with the method I outlined in OP and it is sometimes useful.
  - kaydub 1y ago
    > (though it can help with that too) Yeah, this is a big thing I'm noticing a lot of people miss. I have tons of people ask me "how do I get claude to do <whatever>?" "Ask claude" is the only response I can give. You can get the LLM to help you figure out how to get to your goal and write the right prompt before you even ask the LLM to get to your goal.
- m_fayer 1y ago
  I can imagine it often being the case that if you measure a concise moderately difficult task over half a day or a few days, coding by hand might be faster. But I think, and this is just conjecture, that if you measure over a longer timespan, the ai assisted route will be consistently faster. And for me, this is down to momentum and stamina. Paired with the ai, I’m much more forward looking, always anticipating the next architectural challenge and filling in upcoming knowledge and resource gaps. Without the ai, I would be expending much more energy on managing people and writing code myself. I would be much more stop-and-start as I pause, take stock, deal with human and team issues, and rebuild my capacity for difficult abstract thinking. Paired with a good ai agent and if I consistently avoid the well known pitfalls of said agent, development feels like it has the pace of cross country skiing, a long pleasant steady and satisfying burn.
- bentt 1y ago
  The truth of it is that when I code with an LLM I scope the work up to include parts that would be a stretch for me to implement. I know what I want them to do, I know where I could find the info to write the code, but the LLM can just spit it out and if it's validate-able, then great, on to the next. If I were to attack the same system myself without any LLM assist, I'd make a lot of choices to optimize for my speed and knowledge base. The code would end up much simpler. For something that would be handed off to another person (including future me) that can be a win. But if the system is self contained then going bigger and fancier in that moment can be a win. It all depends on the exact goals. All in all, there's a lot of nuance to this stuff and it's probably not really replacing anyone except people who A) aren't that skilled to start with and B) spend more time yelling about how bad AI is than actually digging in and trying stuff.
Hendrikto 1y ago
> I could have written it but it would have taken me about 3x longer when all is said and done. Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself. Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not?
- giantg2 1y ago
  "Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself." And if this is true, you will have to coach AI each time whereas a person should advance over time.
  - rolisz 1y ago
    And then they skip to another job for more money, and you start again with a new hire.
    - Avicebron 1y ago
      Thankfully after many generations of human interactions and complex analysis of group dynamics, we've found a solution. It's called 'don't be an asshole' and 'pay people competitively'. edit: because people are stupid, 'competitively' in this sense isn't some theoretical number pulled from an average, it's 'does this person feel better off financially working with you than others around them who don't work with you, and is is this person meeting their own personal financial goals through working with you'?
      - wiseowise 1y ago
        They do have a point. Why waste a time on person who will always need more money over time, rather than invest in AI? Not only you don’t need to please every hire, your seniors will be more thankful too, because they will get linearly faster with time.
        1y ago
raphinou 1y ago
I usually ask it to build a feature based on a specification I wrote. If it is not exactly right, it is often the case that editing it myself is faster than iterating with the ai, which has sometimes put me in an infinite loop of corrections requests. Have you encountered this too?
- notarobot123 1y ago
  Have you tried a more granular strategy - smaller chunks and more iterative cycles?
  - swat535 1y ago
    At that point, you might as well write it yourself. Instead of writing 300 lines of code, you are writing 300 lines of prompts. What benefit would you get?
    - andoando 1y ago
      Its not. "Add this table, write the dto" takes 10 seconds to do. It would take me few mins probably assuming Im familiar with the language and much longer if Im not. But its a lot better than that. "Write this table. from here store it into table. Write endpoint to return all from the table" I also had good luck with stuff like "scrape this page, collect x and y, download link pointed at y, store in this directory".
- prox 1y ago
  For me I only use it as a second opinion, I got a pretty good idea of what I want and how to do it, and I can ask any input on what I have written. This gives me the best results sofar.
- pdimitar 1y ago
  This only happens if you want it to one-shot stuff, or if you fall under the false belief that "it is so close, we just need to correct these three things!". Yes I have encountered it. Narrowing focus and putting constraints and guiding it closer made the LLM agent much better at producing what I need. It boils down to me not writing the code really. Using LLMs actually sharpened my architectural and software design skills. Made me think harder and deeper at an earlier stage.
chasd00 1y ago
I’ve found LLMs are most useful when I know what I want to do but just don’t want to type it all out. My best success so far was an LLM saving me about 1,000 lines of typing and fixing syntax mistakes on a web component plus backend in a proprietary framework.
- perrygeo 1y ago
  Yep, and the productivity of LLMs means that experienced developers can go from idea to implementation way faster. But first someone has to understand and define a solid structure. And later someone needs to review, test, and integrate the code into this framework. This is hard stuff. Arguably harder than writing code in the first place! It's no wonder inexperienced developers don't get as much out of it. They define a vague structure, full of problems, but the sycophantic AI will spew out conformant code anyways. Garbage in, garbage out. Bad ideas + fast code gen is just not very productive in the long term - LLMs have made the quality of ideas matter again!
  - esafak 1y ago
    It takes experience to be able to push back.
celticninja 1y ago
A junior would see the solution works and create a PR. A senior knows it works, why it works and what can be improved, then they open a PR. AI is great at a first draft of anything, code, images, text, but the real skill is turning that first draft into something good.
- DarkNova6 1y ago
  I don't see this a problem of seniority but one of mindset. I've met enough "senior devs" that will push just about anything and curious juniors that are much more selective about their working process.
  - glouwbug 1y ago
    In the age of high interest rates everyone is pushing quantity over quality
    - DarkNova6 1y ago
      I fail to see the causality.
      - glouwbug 1y ago
        High interest rates bring layoffs. Layoffs require performance, or at least perceived performance
  - skydhash 1y ago
    I believe senior here means experienced, not older.
- WalterSear 1y ago
  IMHO, not really, if you know what you want. There will always be small things to fix, but if there needs to be a second draft, I would hazard that the PR was too big all along: a problem whether an AI is involved or not.
fauigerzigerk 1y ago
>... it doesn't matter so much that you know every detail of the implementation, as long as it is validated. What makes me nervous is when we generate both the implementation and the test cases. In what sense is this validation?
- nerpderp82 1y ago
  Byzantine Incompleteness enters the chat. Either you go formal, or you test the tests, and then test those ...
- zdragnar 1y ago
  My last attempt had passing tests. It even exercised the code that it wrote! But, upon careful inspection, the test assertions were essentially checking that true equalled true, and the errors in the code didn't fail the tests at all. Attempting to guide it to fixing the errors just introduced novel errors that it didn't make the first time around. This is not what I signed up for.
cjonas 1y ago
Ya the early "studies" that said AI would benefit low skill more than senior never seem grounded in reality. Coding with AI is like having a team of juniors that can complete their assignments in a few minutes instead of days. The more clear your instructions, the closer it is to what you wanted, but there are almost always changes needed. Unfortunately it really does make the junior dev position redundant (although this may prove to be very short-sighted when all the SR devs retire).
- dasil003 1y ago
  I think the idea was that LLMs can allow someone who has no idea how to code, to write a prompt that can in fact output some working code. This is greatly raising their skill floor, as opposed to a senior where at best it’s doing something they already can do, just faster. The elephant in the room being that if you aren’t senior enough to have written the code you’ll probably run into a catastrophic bug that you are incapable of fixing (or prompting the LLM to fix) very very quickly. Really it’s just the next iteration of no-code hype where people dream of building apps without code, but then reality always come back to the fact that the essential skill of programmers is to understand and design highly structured and rigid logical systems. Code is just a means of specification. LLMs make it easier to leverage code patterns that have been written over and over by the hundreds of thousands of programmers that have contributed to its training corpus, but they can not replace the precision of thought needed to make a hand-wavy idea into a concrete system that actually behaves in a way that humans find useful.
- flustercan 1y ago
  I've never worked anywhere where the role of a Sr was to glue together a bunch of small pieces written by a team of Jr devs. I've only worked places where Jr's were given roughly the same scope of work as a mid-level dev but on non-critical projects where they could take as much time as necessary and where mistakes would have a very small blast radius. That type of Jr work has not been made redundant - although I suppose now its possible for a PM or designer to do that work instead (but if your PMs are providing more value by vibe coding non-critical features than by doing their PM work maybe you don't really need a PM?)
hulitu 1y ago
> So in the end, it's code that I know very, very well. I could have written it but it would have taken me about 3x longer when all is said and done. What about the bugs ? Whould you have inserted the same bugs or different ones ?
hoppp 1y ago
Sounds like its faster to just write the code by hand
- glouwbug 1y ago
  The massive productivity gains I’ve seen come from multidisciplinary approaches, where you’d be applying science and engineering from fields like chemistry, physics, thermodynamics, fluids, etc, to speedy compiled languages. The output is immediately verifiable with a bit of trial and error and visualization and you’re saved literally months of up front text book and white paper research to start prototyping anything
- bentt 1y ago
  Once you get a sense for LLM workflow, sometimes the task is not appropriate for it and you do write by hand. In fact, most code I write is by hand. But if I want a new system and the specs are clear, it can be built up in stages that are testable, and there are bits that would take some research but are well documented… then it can be a win. The studies that say devs are slower with LLMs is fair because on average, devs don’t know how to optimize for them. Some do though.
- paool 1y ago
  Even if that's true today ( it's not ), it becomes less true over time as tools and models improve. If you have someone who knows what they're doing with the latest and greatest coding agents it's just not comparable. You can have a Dev open up four or more terminals with multiple prompts running at the same time. A manual person just can't keep up.
  - hoppp 1y ago
    Yeah but then all the code is unaudited? Its too much to review... I generate code but I still place it there by hand, vibe code is just too full of bugs and I find it a bit boring to wait for code to generate all the time.
cmiles74 1y ago
I can see how this workflow made the senior developer faster. At the same time, work mentoring the AI strikes me as less valuable then the same time spent mentoring a junior developer. If this ends up encouraging an ever widening gap between the skill levels of juniors and seniors, I think that would be bad for the field, overall. Getting that kind of data is difficult, right now it's just something I worry about.
- square_usual 1y ago
  > work mentoring the AI strikes me as less valuable then the same time spent mentoring a junior developer But where can you just "mentor" a junior? Hiring people is not so easy, especially not ones that are worth mentoring. Not every junior will be a willing, good recipient of mentoring, and that's if you manage to get one, given budget constraints and long lead times on hiring. And at best you end up with one or two; with parallel LLMs, you can have almost entire teams of people working for you. I'm not arguing for replacing juniors - I worry about the same thing you do - but I can see why companies are so eager to use AI, especially smaller startups that don't have the budgets and manpower to hire people.
  - AdrianB1 1y ago
    If a junior is not willing to learn and grow, there is no future for that person in the organization. "Forever junior" is not a valid job title. Better not hire someone that is not good enough than having to deal with the consequences, I learned from my past mistakes.
    - square_usual 1y ago
      Of course, and that's why it's not a simple choice between using AI and hiring a junior. Hiring and mentoring a junior is a lot more work for an uncertain payoff.
- sosborn 1y ago
  It's a classic short-term gain outlook for these companies.
leemoore 1y ago
My success and experience generally matches yours (and the authors'). Based on my experience over the last 6 months, nothing here around more senior developers getting more productivity and why is remotely controversial. It's fascinating how a report like yours or theirs acts as a lightning rod for those who either haven't been able to work it out or have rigid mental models about how AI doesn't work and want to disprove the experience of those who choose to share their success. A couple of points I'd add to these observations: Even if AI didn't speed anything up... even if it slowed me down by 20%, what I find is that the mental load of coding is reduced in a way that allows me to code for far more hours in a day. I can multitask, attend meetings, get 15 minutes to work on a coding task, and push it forward with minimal coding context reload tax. Just the ability to context switch in and out of coding, combined with the reduced cognitive effort, would still increase my productivity because it allows me to code productively for many more hours per week with less mental fatigue. But on top of that, I also antectodally experience the 2-5x speedup depending on the project. Occasionally things get difficult and maybe I only get a 1.2-1.5x speedup. But it's far easier to slot many more coding hours into the week as an experienced tech lead. I'm leaning far more on skills that are fast, intuitive abilities built up from natural talent and decades of experience: system design, technical design, design review, code review, sequencing dependencies, parsing and organizing work. Get all these things to a high degree of correctness and the coding goes much smoother, AI or no AI. AI gets me through all of these faster, outputs clear curated (by me) artifacts, and does the coding faster. What doesn't get discussed enough is that effective AI-assisted coding has a very high skill ceiling, and there are meta-skills that make you better from the jump: knowing what you want while also having cognitive flexibility to admit when you're wrong; having that thing you want generally be pretty close to solid/decent/workable/correct (some mixture of good judgement & wisdom); communicating well; understanding the cognitive capabilities of humans and human-like entities; understanding what kind of work this particular human/human-like entity can and should do; understanding how to sequence and break down work; having a feel for what's right and wrong in design and code; having an instinct for well-formed requirements and being able to articulate why when they aren't well-formed and what is needed to make them well-formed. These are medium and soft skills that often build up in experienced tech leads and senior developers. This is why it seems that experienced tech leads and senior developers embracing this technology are coming out of the gate with the most productivity gains. I see the same thing with young developers who have a talent for system design, good people-reading skills, and communication. Those with cognitive flexibility and the ability to be creative in design, planning and parsing of work. This isn't your average developer, but those with these skills have much more initial success with AI whether they are young or old. And when you have real success with AI, you get quite excited to build on that success. Momentum builds up which starts building those learning skill hours. Do you need all these meta-skills to be successful with AI? No, but if you don't have many of them, it will take much longer to build sufficient skill in AI coding for it to gain momentum—unless we find the right general process that folks who don't have a natural talent for it can use to be successful. There's a lot going on here with folks who take to AI coding and folks who dont. But it's not terribly surprising that it's the senior devs and old tech leads who tend to take to it faster.
risyachka 1y ago
>> llms where supposed to help juniors Lol what Who came up with this? They never were supposed to do anything. Just turned out to be useful in experienced hands as expected
andrewgleave 1y ago
Yes. Juniors have a lack of knowledge about how to build coherent mental models of problems whose solution will ultimately be implemented in code, whereas seasoned engineers do. Seniors can make this explicit to models and use them to automate "the code they would have written," whereas a junior doesn’t know what they would have written nor how they would have solved it absent a LLM. Same applies to all fields: LLMs can be either huge leverage on top of existing knowledge or a crutch for a lack of understanding.
JumpCrisscross 1y ago
The best way I’ve come to describe LLMs is as an ambitious, occasionally bewilderingly stupid but always incredibly hard working junior employee. You have to watch what it’s doing. And you can’t let it take out into territory you don’t understand, because it will fuck up off leash. But it will thanklessly iterate revision after revision in the way one would ordinarily do with a team, but now don’t need to for tasks that would bore them.
- s1artibartfast 1y ago
  My mental model is a very smart and interesting stranger at a bar. Sometimes detail accuracy is sacrificed in service of a good story. Sometimes they simply full of shit and double down when pushed.
somethingsome 1y ago
Aaaha for me it was exactly the other way around! I had a very complex piece of logic, with many many many moving parts. So I implemented many paths of that logic by hand, every one with their own specifics. Every path took something like 200-400 lines of code. When this was done and correct. It was difficult to see the moving parts through the forest. Some code was similar but still a bit different, and hard to think about and spread in many places. I put everything into an Llm and asked about isolation, architecture and refactoring. And it actually gave me pretty good abstractions and a good architecture. It didn't include every possible path, but was easy enough to be continued by hand. It's not that I would not have tought about it. But it was easier that way, and probably my handcrafted solution would be very similar (+headache). Of course, I reviewed it extensively, and reimplemented every missing path and corrected the ones that were buggy now by hand. For the experiment, I played with agents to fill the missing parts, it was a disaster :)
- bentt 1y ago
  Ah this is great, and I can see how that would be useful. In a way, it's just "there's a clear spec" by another path, but it's just defined by all the code you wrote.

by bentt 1y ago

mattmanser 1y ago

Would it have actually taken you 3x longer? I am surprising myself these days with how fast I'm being using AI as a glorified Stack Overflow. We are also having studies and posts come out that when actually tried side-by-side, the AI writes the coding route is slower, though the developer percieves it as faster.

WesolyKubeczek 1y ago
> the AI writes the coding route is slower, though the developer percieves it as faster. I have this pattern while driving. Using the main roads, when there is little to no traffic, the commute is objectively, measurably the fastest. However, during peak hours, I find myself in traffic jams, so I divert to squiggly country roads which are both slower and longer, but at least I’m moving all the time. The thing is, when I did have to take the main road during the peak traffic, the difference between it and squiggly country roads was like two to three minutes at worst, and not half an hour like I was afraid it would be. Sure, ten minutes crawling or standing felt like an hour. Maybe coding with LLMs makes you think you are doing something productive the whole time, but the actual output is little different from the old way? But hey, at least it’s not like you’re twiddling your thumbs for hours, and the bossware measuring your productivity by your keyboard and mouse activity is happy!
- realisticfoo 1y ago
  the best use of LLM is to help you program a digispark USB dongle with Arduino to make it emulate mouse movement and scrolling to appease the idi0tic spyware retarded boomers monitor like it was relevant somehow on their corporate spy box. Meanwhile we adults can do real work on a separate real computer. Never use their laptop more than absolutely minimum possible.
  - WesolyKubeczek 1y ago
    Well, you might have several obstacles here: 1) the bossware might take screenshots too 2) the bosses pay for the whole LLM so they expect you to use the whole LLM 3) you may not want to contaminate your spare computer with whatever shit you're working on on job, and indeed it may be considered a breach of security (as if feeding OpenAI/Anthropic isn't, lol, but that's beside the point. So you continue to feel miserable, but you get your paycheck, and it's better than unemployment, and your kids are fed and clothed, so there's that.
notarobot123 1y ago
I am not the biggest fan of LLMs but I have to admit that, as long as you understand what the technology is and how it works, it is a very powerful tool. I think the mixed reports on utility have a lot to do with the very different ways the tool is used and how much 'magic' the end-user expects versus how much the end-user expects to guide the tool to do the work. To get the best out of it, you do have to provide significant amount of scaffolding (though it can help with that too). If you're just pointing it at a codebase and expecting it to figure it out, you're going to have mixed results at best. If you guide it well, it can save a significant amount of manual effort and time.
- bentt 1y ago
  yeah very few months I try to have it “just do magic” again and I re-learn the lesson. Like, I’ll just say “optimize this shader!” and plug it in blind. It doesn’t work. The only way it could is if the LLM has a testing loop itself. I guess in web world it could, but in my world of game dev, not so much. So I stick with the method I outlined in OP and it is sometimes useful.
- kaydub 1y ago
  > (though it can help with that too) Yeah, this is a big thing I'm noticing a lot of people miss. I have tons of people ask me "how do I get claude to do <whatever>?" "Ask claude" is the only response I can give. You can get the LLM to help you figure out how to get to your goal and write the right prompt before you even ask the LLM to get to your goal.
m_fayer 1y ago
I can imagine it often being the case that if you measure a concise moderately difficult task over half a day or a few days, coding by hand might be faster. But I think, and this is just conjecture, that if you measure over a longer timespan, the ai assisted route will be consistently faster. And for me, this is down to momentum and stamina. Paired with the ai, I’m much more forward looking, always anticipating the next architectural challenge and filling in upcoming knowledge and resource gaps. Without the ai, I would be expending much more energy on managing people and writing code myself. I would be much more stop-and-start as I pause, take stock, deal with human and team issues, and rebuild my capacity for difficult abstract thinking. Paired with a good ai agent and if I consistently avoid the well known pitfalls of said agent, development feels like it has the pace of cross country skiing, a long pleasant steady and satisfying burn.
bentt 1y ago
The truth of it is that when I code with an LLM I scope the work up to include parts that would be a stretch for me to implement. I know what I want them to do, I know where I could find the info to write the code, but the LLM can just spit it out and if it's validate-able, then great, on to the next. If I were to attack the same system myself without any LLM assist, I'd make a lot of choices to optimize for my speed and knowledge base. The code would end up much simpler. For something that would be handed off to another person (including future me) that can be a win. But if the system is self contained then going bigger and fancier in that moment can be a win. It all depends on the exact goals. All in all, there's a lot of nuance to this stuff and it's probably not really replacing anyone except people who A) aren't that skilled to start with and B) spend more time yelling about how bad AI is than actually digging in and trying stuff.

Hendrikto 1y ago

> I could have written it but it would have taken me about 3x longer when all is said and done. Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself. Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not?

giantg2 1y ago
"Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself." And if this is true, you will have to coach AI each time whereas a person should advance over time.
- rolisz 1y ago
  And then they skip to another job for more money, and you start again with a new hire.
  - Avicebron 1y ago
    Thankfully after many generations of human interactions and complex analysis of group dynamics, we've found a solution. It's called 'don't be an asshole' and 'pay people competitively'. edit: because people are stupid, 'competitively' in this sense isn't some theoretical number pulled from an average, it's 'does this person feel better off financially working with you than others around them who don't work with you, and is is this person meeting their own personal financial goals through working with you'?
    - wiseowise 1y ago
      They do have a point. Why waste a time on person who will always need more money over time, rather than invest in AI? Not only you don’t need to please every hire, your seniors will be more thankful too, because they will get linearly faster with time.
      - 1y ago

raphinou 1y ago

I usually ask it to build a feature based on a specification I wrote. If it is not exactly right, it is often the case that editing it myself is faster than iterating with the ai, which has sometimes put me in an infinite loop of corrections requests. Have you encountered this too?

notarobot123 1y ago
Have you tried a more granular strategy - smaller chunks and more iterative cycles?
- swat535 1y ago
  At that point, you might as well write it yourself. Instead of writing 300 lines of code, you are writing 300 lines of prompts. What benefit would you get?
  - andoando 1y ago
    Its not. "Add this table, write the dto" takes 10 seconds to do. It would take me few mins probably assuming Im familiar with the language and much longer if Im not. But its a lot better than that. "Write this table. from here store it into table. Write endpoint to return all from the table" I also had good luck with stuff like "scrape this page, collect x and y, download link pointed at y, store in this directory".
prox 1y ago
For me I only use it as a second opinion, I got a pretty good idea of what I want and how to do it, and I can ask any input on what I have written. This gives me the best results sofar.
pdimitar 1y ago
This only happens if you want it to one-shot stuff, or if you fall under the false belief that "it is so close, we just need to correct these three things!". Yes I have encountered it. Narrowing focus and putting constraints and guiding it closer made the LLM agent much better at producing what I need. It boils down to me not writing the code really. Using LLMs actually sharpened my architectural and software design skills. Made me think harder and deeper at an earlier stage.

chasd00 1y ago

I’ve found LLMs are most useful when I know what I want to do but just don’t want to type it all out. My best success so far was an LLM saving me about 1,000 lines of typing and fixing syntax mistakes on a web component plus backend in a proprietary framework.

perrygeo 1y ago
Yep, and the productivity of LLMs means that experienced developers can go from idea to implementation way faster. But first someone has to understand and define a solid structure. And later someone needs to review, test, and integrate the code into this framework. This is hard stuff. Arguably harder than writing code in the first place! It's no wonder inexperienced developers don't get as much out of it. They define a vague structure, full of problems, but the sycophantic AI will spew out conformant code anyways. Garbage in, garbage out. Bad ideas + fast code gen is just not very productive in the long term - LLMs have made the quality of ideas matter again!
- esafak 1y ago
  It takes experience to be able to push back.

celticninja 1y ago

A junior would see the solution works and create a PR. A senior knows it works, why it works and what can be improved, then they open a PR. AI is great at a first draft of anything, code, images, text, but the real skill is turning that first draft into something good.

DarkNova6 1y ago
I don't see this a problem of seniority but one of mindset. I've met enough "senior devs" that will push just about anything and curious juniors that are much more selective about their working process.
- glouwbug 1y ago
  In the age of high interest rates everyone is pushing quantity over quality
  - DarkNova6 1y ago
    I fail to see the causality.
    - glouwbug 1y ago
      High interest rates bring layoffs. Layoffs require performance, or at least perceived performance
- skydhash 1y ago
  I believe senior here means experienced, not older.
WalterSear 1y ago
IMHO, not really, if you know what you want. There will always be small things to fix, but if there needs to be a second draft, I would hazard that the PR was too big all along: a problem whether an AI is involved or not.

fauigerzigerk 1y ago

>... it doesn't matter so much that you know every detail of the implementation, as long as it is validated. What makes me nervous is when we generate both the implementation and the test cases. In what sense is this validation?

nerpderp82 1y ago
Byzantine Incompleteness enters the chat. Either you go formal, or you test the tests, and then test those ...
zdragnar 1y ago
My last attempt had passing tests. It even exercised the code that it wrote! But, upon careful inspection, the test assertions were essentially checking that true equalled true, and the errors in the code didn't fail the tests at all. Attempting to guide it to fixing the errors just introduced novel errors that it didn't make the first time around. This is not what I signed up for.

cjonas 1y ago

Ya the early "studies" that said AI would benefit low skill more than senior never seem grounded in reality. Coding with AI is like having a team of juniors that can complete their assignments in a few minutes instead of days. The more clear your instructions, the closer it is to what you wanted, but there are almost always changes needed. Unfortunately it really does make the junior dev position redundant (although this may prove to be very short-sighted when all the SR devs retire).

dasil003 1y ago
I think the idea was that LLMs can allow someone who has no idea how to code, to write a prompt that can in fact output some working code. This is greatly raising their skill floor, as opposed to a senior where at best it’s doing something they already can do, just faster. The elephant in the room being that if you aren’t senior enough to have written the code you’ll probably run into a catastrophic bug that you are incapable of fixing (or prompting the LLM to fix) very very quickly. Really it’s just the next iteration of no-code hype where people dream of building apps without code, but then reality always come back to the fact that the essential skill of programmers is to understand and design highly structured and rigid logical systems. Code is just a means of specification. LLMs make it easier to leverage code patterns that have been written over and over by the hundreds of thousands of programmers that have contributed to its training corpus, but they can not replace the precision of thought needed to make a hand-wavy idea into a concrete system that actually behaves in a way that humans find useful.
flustercan 1y ago
I've never worked anywhere where the role of a Sr was to glue together a bunch of small pieces written by a team of Jr devs. I've only worked places where Jr's were given roughly the same scope of work as a mid-level dev but on non-critical projects where they could take as much time as necessary and where mistakes would have a very small blast radius. That type of Jr work has not been made redundant - although I suppose now its possible for a PM or designer to do that work instead (but if your PMs are providing more value by vibe coding non-critical features than by doing their PM work maybe you don't really need a PM?)

hulitu 1y ago

> So in the end, it's code that I know very, very well. I could have written it but it would have taken me about 3x longer when all is said and done. What about the bugs ? Whould you have inserted the same bugs or different ones ?

hoppp 1y ago

Sounds like its faster to just write the code by hand

glouwbug 1y ago
The massive productivity gains I’ve seen come from multidisciplinary approaches, where you’d be applying science and engineering from fields like chemistry, physics, thermodynamics, fluids, etc, to speedy compiled languages. The output is immediately verifiable with a bit of trial and error and visualization and you’re saved literally months of up front text book and white paper research to start prototyping anything
bentt 1y ago
Once you get a sense for LLM workflow, sometimes the task is not appropriate for it and you do write by hand. In fact, most code I write is by hand. But if I want a new system and the specs are clear, it can be built up in stages that are testable, and there are bits that would take some research but are well documented… then it can be a win. The studies that say devs are slower with LLMs is fair because on average, devs don’t know how to optimize for them. Some do though.
paool 1y ago
Even if that's true today ( it's not ), it becomes less true over time as tools and models improve. If you have someone who knows what they're doing with the latest and greatest coding agents it's just not comparable. You can have a Dev open up four or more terminals with multiple prompts running at the same time. A manual person just can't keep up.
- hoppp 1y ago
  Yeah but then all the code is unaudited? Its too much to review... I generate code but I still place it there by hand, vibe code is just too full of bugs and I find it a bit boring to wait for code to generate all the time.

cmiles74 1y ago

I can see how this workflow made the senior developer faster. At the same time, work mentoring the AI strikes me as less valuable then the same time spent mentoring a junior developer. If this ends up encouraging an ever widening gap between the skill levels of juniors and seniors, I think that would be bad for the field, overall. Getting that kind of data is difficult, right now it's just something I worry about.

square_usual 1y ago
> work mentoring the AI strikes me as less valuable then the same time spent mentoring a junior developer But where can you just "mentor" a junior? Hiring people is not so easy, especially not ones that are worth mentoring. Not every junior will be a willing, good recipient of mentoring, and that's if you manage to get one, given budget constraints and long lead times on hiring. And at best you end up with one or two; with parallel LLMs, you can have almost entire teams of people working for you. I'm not arguing for replacing juniors - I worry about the same thing you do - but I can see why companies are so eager to use AI, especially smaller startups that don't have the budgets and manpower to hire people.
- AdrianB1 1y ago
  If a junior is not willing to learn and grow, there is no future for that person in the organization. "Forever junior" is not a valid job title. Better not hire someone that is not good enough than having to deal with the consequences, I learned from my past mistakes.
  - square_usual 1y ago
    Of course, and that's why it's not a simple choice between using AI and hiring a junior. Hiring and mentoring a junior is a lot more work for an uncertain payoff.
sosborn 1y ago
It's a classic short-term gain outlook for these companies.

leemoore 1y ago

My success and experience generally matches yours (and the authors'). Based on my experience over the last 6 months, nothing here around more senior developers getting more productivity and why is remotely controversial. It's fascinating how a report like yours or theirs acts as a lightning rod for those who either haven't been able to work it out or have rigid mental models about how AI doesn't work and want to disprove the experience of those who choose to share their success. A couple of points I'd add to these observations: Even if AI didn't speed anything up... even if it slowed me down by 20%, what I find is that the mental load of coding is reduced in a way that allows me to code for far more hours in a day. I can multitask, attend meetings, get 15 minutes to work on a coding task, and push it forward with minimal coding context reload tax. Just the ability to context switch in and out of coding, combined with the reduced cognitive effort, would still increase my productivity because it allows me to code productively for many more hours per week with less mental fatigue. But on top of that, I also antectodally experience the 2-5x speedup depending on the project. Occasionally things get difficult and maybe I only get a 1.2-1.5x speedup. But it's far easier to slot many more coding hours into the week as an experienced tech lead. I'm leaning far more on skills that are fast, intuitive abilities built up from natural talent and decades of experience: system design, technical design, design review, code review, sequencing dependencies, parsing and organizing work. Get all these things to a high degree of correctness and the coding goes much smoother, AI or no AI. AI gets me through all of these faster, outputs clear curated (by me) artifacts, and does the coding faster. What doesn't get discussed enough is that effective AI-assisted coding has a very high skill ceiling, and there are meta-skills that make you better from the jump: knowing what you want while also having cognitive flexibility to admit when you're wrong; having that thing you want generally be pretty close to solid/decent/workable/correct (some mixture of good judgement & wisdom); communicating well; understanding the cognitive capabilities of humans and human-like entities; understanding what kind of work this particular human/human-like entity can and should do; understanding how to sequence and break down work; having a feel for what's right and wrong in design and code; having an instinct for well-formed requirements and being able to articulate why when they aren't well-formed and what is needed to make them well-formed. These are medium and soft skills that often build up in experienced tech leads and senior developers. This is why it seems that experienced tech leads and senior developers embracing this technology are coming out of the gate with the most productivity gains. I see the same thing with young developers who have a talent for system design, good people-reading skills, and communication. Those with cognitive flexibility and the ability to be creative in design, planning and parsing of work. This isn't your average developer, but those with these skills have much more initial success with AI whether they are young or old. And when you have real success with AI, you get quite excited to build on that success. Momentum builds up which starts building those learning skill hours. Do you need all these meta-skills to be successful with AI? No, but if you don't have many of them, it will take much longer to build sufficient skill in AI coding for it to gain momentum—unless we find the right general process that folks who don't have a natural talent for it can use to be successful. There's a lot going on here with folks who take to AI coding and folks who dont. But it's not terribly surprising that it's the senior devs and old tech leads who tend to take to it faster.

risyachka 1y ago

>> llms where supposed to help juniors Lol what Who came up with this? They never were supposed to do anything. Just turned out to be useful in experienced hands as expected

andrewgleave 1y ago

Yes. Juniors have a lack of knowledge about how to build coherent mental models of problems whose solution will ultimately be implemented in code, whereas seasoned engineers do. Seniors can make this explicit to models and use them to automate "the code they would have written," whereas a junior doesn’t know what they would have written nor how they would have solved it absent a LLM. Same applies to all fields: LLMs can be either huge leverage on top of existing knowledge or a crutch for a lack of understanding.

JumpCrisscross 1y ago

The best way I’ve come to describe LLMs is as an ambitious, occasionally bewilderingly stupid but always incredibly hard working junior employee. You have to watch what it’s doing. And you can’t let it take out into territory you don’t understand, because it will fuck up off leash. But it will thanklessly iterate revision after revision in the way one would ordinarily do with a team, but now don’t need to for tasks that would bore them.

s1artibartfast 1y ago
My mental model is a very smart and interesting stranger at a bar. Sometimes detail accuracy is sacrificed in service of a good story. Sometimes they simply full of shit and double down when pushed.

somethingsome 1y ago

Aaaha for me it was exactly the other way around! I had a very complex piece of logic, with many many many moving parts. So I implemented many paths of that logic by hand, every one with their own specifics. Every path took something like 200-400 lines of code. When this was done and correct. It was difficult to see the moving parts through the forest. Some code was similar but still a bit different, and hard to think about and spread in many places. I put everything into an Llm and asked about isolation, architecture and refactoring. And it actually gave me pretty good abstractions and a good architecture. It didn't include every possible path, but was easy enough to be continued by hand. It's not that I would not have tought about it. But it was easier that way, and probably my handcrafted solution would be very similar (+headache). Of course, I reviewed it extensively, and reimplemented every missing path and corrected the ones that were buggy now by hand. For the experiment, I played with agents to fill the missing parts, it was a disaster :)

bentt 1y ago
Ah this is great, and I can see how that would be useful. In a way, it's just "there's a clear spec" by another path, but it's just defined by all the code you wrote.