3 ms·

Yes! I don't mind it so much when it's a newbie or non-techie who has never actually written code before, because bless their hearts, they did it! They got som

Yes! I don't mind it so much when it's a newbie or non-techie who has never actually written code before, because bless their hearts, they did it! They got some code working! But if you've been developing for decades, you know that counting lines of code means nothing, less than nothing. That you could probably achieve the same result in half the lines if you thought about it a bit longer. And to claim this as an achievement when it's LLM-generated... that's not a boast. That doesn't mean what you think it means. But I guess we hit the same old problem that we've always had - how do you measure productivity in software development? If you wanted to boast about how an LLM is making you 100x more productive, what metric could you use? LOC is the most easily measurable, really, really, terrible measure that PMs have been using since we started doing this, because everything else is hard.

eucyclos 6mo ago
I forget who said it, but I heard the idea floated that if your work can be measured in terms of productivity at all, it can and probably should be done by software. Not sure how that applies here since as you point out, a 10x programmer probably doesn't produce 10x the code.
bdangubic 6mo ago
here’s one thing that somewhat worked for my team. when we first started using LLMs we decided to run the same process as if they did not exist, same sprint planning meetings, same estimation. we did this for 6 months and saw roughly 55% increase in output compared to pre-LLM usage. there are biases in what were tried to achieve, it is not easy to estimate something will take XX hours when you know some portion (for example writing documentation or portions of the test coverage) you won’t have to write but we did our best. after we convinced ourselves of productivity gains we stopped doing this.
- marcus_holmes 6mo ago
  wow, great experiment. I'm amazed the whole team went through with duplicating everything for that long. Nice work :) I resorted to feels. After decades of programming, I know when I'm being productive, and I can reasonably estimate when a colleague is being productive. I extrapolate that to the LLM, too. Absolutely not an objective measure, but I feel that I can get the LLM to do in a day a task that would take me 2-3 weeks (post-Nov 25 and using parallel agents).
  - bdangubic 6mo ago
    I did too also generally resorted to feels. However, as a team we decided that we need to convince ourselves that LLMs are both an accelerator as well as not introducing flaws in our process (regressions, performance issues, security issues...). My team is pretty awesome and we generally try not to "fall for FOMO" type deals, like "everyone is using ____, surely we must as well." just like HN, we had team members that were hesitant to say the least in the beginning as well as team members that were "convinced" a lot earlier that LLMs can be a great accelerator to what we are doing. I would venture a guess that similar situation exists(ed) in many places. so we tried to figure out a way where it is not just portion of the team "putting the foot down" so-to-speak but more like "OK, lets see if this can be measured so that everyone (or lets say overwhelming majority) is on board." I wish I would read that more teams at least attempted this approach...

eucyclos 6mo ago

I forget who said it, but I heard the idea floated that if your work can be measured in terms of productivity at all, it can and probably should be done by software. Not sure how that applies here since as you point out, a 10x programmer probably doesn't produce 10x the code.

bdangubic 6mo ago

here’s one thing that somewhat worked for my team. when we first started using LLMs we decided to run the same process as if they did not exist, same sprint planning meetings, same estimation. we did this for 6 months and saw roughly 55% increase in output compared to pre-LLM usage. there are biases in what were tried to achieve, it is not easy to estimate something will take XX hours when you know some portion (for example writing documentation or portions of the test coverage) you won’t have to write but we did our best. after we convinced ourselves of productivity gains we stopped doing this.

marcus_holmes 6mo ago
wow, great experiment. I'm amazed the whole team went through with duplicating everything for that long. Nice work :) I resorted to feels. After decades of programming, I know when I'm being productive, and I can reasonably estimate when a colleague is being productive. I extrapolate that to the LLM, too. Absolutely not an objective measure, but I feel that I can get the LLM to do in a day a task that would take me 2-3 weeks (post-Nov 25 and using parallel agents).
- bdangubic 6mo ago
  I did too also generally resorted to feels. However, as a team we decided that we need to convince ourselves that LLMs are both an accelerator as well as not introducing flaws in our process (regressions, performance issues, security issues...). My team is pretty awesome and we generally try not to "fall for FOMO" type deals, like "everyone is using ____, surely we must as well." just like HN, we had team members that were hesitant to say the least in the beginning as well as team members that were "convinced" a lot earlier that LLMs can be a great accelerator to what we are doing. I would venture a guess that similar situation exists(ed) in many places. so we tried to figure out a way where it is not just portion of the team "putting the foot down" so-to-speak but more like "OK, lets see if this can be measured so that everyone (or lets say overwhelming majority) is on board." I wish I would read that more teams at least attempted this approach...