4 ms·

I'm using LLMs to assist my development and I'm measurably (in all the ways we engineers could possibly care about) doing better work faster. Studies sugge

I'm using LLMs to assist my development and I'm measurably (in all the ways we engineers could possibly care about) doing better work faster. Studies suggest you aren't any faster and may in fact be slower. It's difficult to study such a new tech, but even optimistically, empirical evidence is only showing a ~3% gain in some domains. Writing code is rarely the limiting factor in our work.

cdelsolar 5mo ago
studies suggest nothing. i've released a massive number of features in the last year for several projects that i estimate would have taken me multiple years to put together in a much more mentally exhausting way.
- b40d-48b2-979e 5mo ago
  Your drive-by comments are unwelcome. You can estimate all you want, but the data is collating, and it isn't within your worldview.
  - shard972 5mo ago
    [dead]
  - lanyard-textile 5mo ago
    It is healthy to question results: That's good science. This result wouldn't surprise me if the tooling was limited to, say, copilot :) It would surprise me if it included tooling like Claude Code. Which seems unlikely, given its recency.
  - cdelsolar 5mo ago
    I am not estimating anything. This is my own evidence from my own projects. I have deployed gigantic features this past year, each of which would have taken me the whole year to write.
adamtaylor_13 5mo ago
Those studies have well known flaws. I'm measuring my output so I happen to know I'm not only going faster, but the quality is better. I'm not vibe-measuring my output ;)
Kiro 5mo ago
There's no uncertainty here. Every day I ask myself how long something I did would have taken without it. The answer is always crystal clear. It's not hard or difficult at all.
solidasparagus 5mo ago
I think those studies have framing or methodological issue. I agree the maintenance burden is probably being undervalued by developers in general, but there's just no way the work I do isn't faster. I just categorically couldn't have achieved the outputs I do now in the time windows I have. The software just wouldn't have existed in the world of 3 years ago and I did enough coding back then to say that with certainty.
tonyedgecombe 5mo ago
Surely that argument is dead once someone has migrated a million lines of code in eight days.
- gregoryl 5mo ago
  Surely that reinforces the argument - there are now a million LOC in a different language, needing stunning amounts of work to validate it actually functions? Writing the code has never been the bottleneck.
  - tonyedgecombe 5mo ago
    Writing code was definitely the roadblock in a rewrite like this.
retromario 5mo ago
In the past 18 months I've seen experienced developers turn out incredible work using llm-assisted tools, over and over again. With the right harnesses, processes and result-oriented testing, you can simply produce so much more high-quality work. I know it's anecdotal, but I have so much data from my own experiences and those of my peers that I know these new tools are here to stay. It also makes me believe that those studies are either flawed or out of date.

cdelsolar 5mo ago

studies suggest nothing. i've released a massive number of features in the last year for several projects that i estimate would have taken me multiple years to put together in a much more mentally exhausting way.

b40d-48b2-979e 5mo ago
Your drive-by comments are unwelcome. You can estimate all you want, but the data is collating, and it isn't within your worldview.
- shard972 5mo ago
  [dead]
- lanyard-textile 5mo ago
  It is healthy to question results: That's good science. This result wouldn't surprise me if the tooling was limited to, say, copilot :) It would surprise me if it included tooling like Claude Code. Which seems unlikely, given its recency.
- cdelsolar 5mo ago
  I am not estimating anything. This is my own evidence from my own projects. I have deployed gigantic features this past year, each of which would have taken me the whole year to write.

adamtaylor_13 5mo ago

Those studies have well known flaws. I'm measuring my output so I happen to know I'm not only going faster, but the quality is better. I'm not vibe-measuring my output ;)

Kiro 5mo ago

There's no uncertainty here. Every day I ask myself how long something I did would have taken without it. The answer is always crystal clear. It's not hard or difficult at all.

solidasparagus 5mo ago

I think those studies have framing or methodological issue. I agree the maintenance burden is probably being undervalued by developers in general, but there's just no way the work I do isn't faster. I just categorically couldn't have achieved the outputs I do now in the time windows I have. The software just wouldn't have existed in the world of 3 years ago and I did enough coding back then to say that with certainty.

tonyedgecombe 5mo ago

Surely that argument is dead once someone has migrated a million lines of code in eight days.

gregoryl 5mo ago
Surely that reinforces the argument - there are now a million LOC in a different language, needing stunning amounts of work to validate it actually functions? Writing the code has never been the bottleneck.
- tonyedgecombe 5mo ago
  Writing code was definitely the roadblock in a rewrite like this.

retromario 5mo ago

In the past 18 months I've seen experienced developers turn out incredible work using llm-assisted tools, over and over again. With the right harnesses, processes and result-oriented testing, you can simply produce so much more high-quality work. I know it's anecdotal, but I have so much data from my own experiences and those of my peers that I know these new tools are here to stay. It also makes me believe that those studies are either flawed or out of date.