6 ms·

The article claims, that senior developers with over 10 years of experience are more than twice as likely to heavily rely on AI tools compared to their junior c

by Rochus 1y ago

The article claims, that senior developers with over 10 years of experience are more than twice as likely to heavily rely on AI tools compared to their junior counterparts. No p-values or statistical significance tests are reported in either The Register article or Fastly's original blog post. I have over 30 years of experience and recently used Claude Opus 4.1 (via browser and claude.ai) to generate an ECMA-335 and an LLVM code generator for a compiler, and a Qt adapter for the Mono soft debugging protocol. Each task resulted in 2-3kLOC of C++. The Claude experience was mixed; there is a high probability that the system doesn't respond or just quickly shows an overloaded message and does nothing. If it generates code, I quckly run in some output limitation and have to manually press "continue", and then often the result gets scrambled (i.e. the order of the generated code fragments gets mixed up, which requires another round with Claude to fix). After this process, the resulting code then compiled immediately, which impressed me. But it is full of omissions and logical errors. I am still testing and correcting. All in all, I can't say at this point that Claude has really taken any work off my hands. In order to understand the code and assess the correctness of the intermediate results, I need to know exactly how to implement the problem myself. And you have to test everything in detail and do a lot of redesigning and correcting. Some implementations are just stubs, and even after several attempts, there was still no implementation. In my opinion, what is currently available (via my $20 subscription) is impressive, but it neither replaces experience nor does it really save time. So yes, now I'm one of the 30% seniors who used AI tools, but I didn't really benefit from them in these specific tasks. Not surprisingly, also the original blog states, that nearly 30% of senior developers report "editing AI output enough to offset most of the time savings". So not really a success so far. But all in all I'm still impressed.

oliwary 1y ago
Hey! I would encourage you to try our Claude code instead, which is also part of your subscription. It's a CLI that takes care of many of the issues you encountered, as it works directly on the code files in a directory. No more copy pasting or unscrambling results. Likewise, it can run commands itself to e.g. compile or even test code.
- Rochus 1y ago
  I'm working on old hardware and not-recent Linux and compiler versions, and I have no confidence yet in allowing AI direct (write) access to my repositories. Instead I provided Claude with the source code of a transpiler to C (one file) which is known to work, uses the same IR as the new code generators were supposed to use. This is a controlled experiment with a clear and complete input and clear expectations and specifications of the output. I don't think I would be able to clearly isolate the contributions and assess the performance of Claude when it has access to arbitrary parts of the source code.
  - kelnos 1y ago
    > I have no confidence yet in allowing AI direct (write) access to my repositories. You don't need to give it write access to your repositories, just to a source tree.
  - torben-friis 1y ago
    On Claude you specifically accept any attempt to use a terminal command (optionally whitelisting) so there’s no risk that it will push force something or whatever. You can also whitelist with granularity, for example to enable it to use git to view git logs but not commit. You can just let it work, see what’s uncommitted after it’s over, and get rid of the result if you don’t like it.
  - stavros 1y ago
    I use Claude Code with the Max plan, and the experience isn't far off from what you describe. You still need to understand the system and review the implementation, because it makes many mistakes. That's not the part it saves me time in, it saves me time in looking up the documentation. Other than that, it might be slower, because the larger the code change is, the more time I need to spend reviewing, and past a point I just can't be bothered. The best way I've found is to have it write small functions, and then I tell it to compose them together. That way, I know exactly what's happening in the code, and I can trust that it works correctly. Cursor is probably a better way to do that than Claude Code, though.
kelnos 1y ago
I hate to play the "you're holding it wrong" card, but when I started, I had more or less the same experience. Eventually you start to learn how to better talk to it in order to get better results. Something I've found useful with Claude Code is that it works a lot better if I give it many small tasks to perform to eventually get the big thing done, rather than just dumping the big thing in its lap. You can do this interactively (prompt, output, prompt, output, prompt, output...) or by writing a big markdown file with the steps to build it laid out.
- Rochus 1y ago
  I just tried to use it for something I consider it to provide the most benefit (for my case). Being able to fully delegate a complicated (and boring) part to a machine would give me more time for the things I'm really interested in. I think we are on the right track in this regard, but we still have a long way to go.
- chillingeffect 1y ago
  Similar here. AI works much better as a consultant than as a developer. I ask it all kinds of things I have suspicions and intuitions about and it provides clarity and examples. It's great for subroutines. Trying to make full programs is just too large of a space. It's difficult to communicate all the implicit requirements.
  - jennyholzer 1y ago
    People who consistently consult LLMs for product direction or software feature design overwhelmingly appear to me as willfully ignorant dullards. I mean it's even further than willful ignorance. It's delight in one's own ignorance.
- JeanMarcS 1y ago
  This. For me (senior as in I am in the field since last century), that's how I use it. "I want to do that, with this data, to obtain that" I still do the part of my job that I got experience on, analyze the need, and use the AI like an assistant to do small libraries or part of code. Like these, errors have less chance to appear. Then I glue that together. For me the time ratio is best use like that. If I have to describe the whole thing, I'm not far from doing it myself, so there's no need for me. Important: I work alone, not in a team, so maybe it has an impact on my thought
epolanski 1y ago
Imho your post summarizes 90% of the posts I see about AI coding on HN: not understanding the tools, not understanding their strenghts and weaknesses, not being good at prompting or context management yet forming strong(ish) opinions. If you don't know what they are good at and how to use them of course you may end up with mixed results and yes, you may waste time. That's a criticism I have also towards AI super enthusiasts (especially vibe coders, albeit you won't find much here), they often confuse the fact that LLMs often one shot 80% of the solutions with the idea that LLMs are 80% there, whereas the Pareto principle well applies to software development where it's the hardest 20% that's gonna prove difficult.
- Rochus 1y ago
  I'm pretty good at prompting and I successfully use Perplexity (mostly with Claude Sonnet 4) to develop concepts, sometimes with the same session expanded over several days. I think the user interface is much superior over Claude.ai. My hope was that the newer Claude Opus 4.1 would be much better in solving complicated coding tasks, which doesn't seem to be the case. For this I had to subscribe to claude.ai. Actually I didn't see much difference in performance, but a much worse UI and availability experience. When it comes to developing a complex topic in a factual dialogue, Claude Sonnet Thinking seems to me to be even more suitable than Claude Opus.
  - epolanski 1y ago
    I'll be more detailed in my second reply. 1) Your original post asks a lot if not too much out the LLM, the expectation you have is too big, to the point that to get anywhere near decent results you need a super detailed prompt (if not several spec documents) and your conclusion stands true: it might be faster to just do it manually. That's the state of LLMs as of today. Your post neither hints at such detailed and laborious prompting nor seem to recognize you've asked it too much, displaying that you are not very comfortable with the limitations of the tool. You're still exploring what it can and what it can't do. But that also implies you're yet not an expert. 2) The second takeaway that you're not yet as comfortable with the tools as you think you are is clearly context management. 2/3k locs of code are way too much. It's a massive amount of output to hope for good results (this also ties with the quality of the prompt, with the guidelines and code practices provided, etc, etc). 3) Neither 1 or 2 are criticisms of your conclusions or opinions, if anything, they are confirmations of your point that LLMs are not there. But what I disagree with is the rush into concluding that AI coding provides net 0 benefits out of your experience. That I don't share. Instead of settling on what it could do (help with planning, writing a spec file, writing unit tests, providing the more boilerplate-y part of the code) and use the LLM to reduce the friction (and thus provide a net benefit), you essentially asked it to replace you and found out the obvious: that LLMs cannot take care of non-trivial business logic yet, and even when they can the results are nowhere near being satisfactory. But that doesn't mean that AI-assisted coding is useless and the net benefit is 0, or negative, it only becomes so as the expectations on the tool are too big and the amount of information provided is either too small to return consistent results or too large for the context to be an issue.
furyofantares 1y ago
I'm just shy of 30 years experience. I think I've spent more time learning how to use these tools than any other technology I've learned, and I still don't know the best way to use them. They certainly weren't a time-saver right away but they became one after some time giving them a real shot. I tested their + my limits on small projects, working out how to get them to do the whole project, figuring out when they stop working and why, figuring out which technology they work best with, figuring out the right size problems to give them, figuring out how to recognize if I'm asking them something they can't do well a ask something different instead, guide them into creating code that they can't actually continue to be successful with. I started last December in Cursor's agentic mode and have been in Claude Code ever since probably March or April. It's definitely been a huge boost all year for side projects - but only in the last couple months have I been having success in a large codebase. Even with all this experience I don't know that I would really be able to get much value out of the chat interface for them. They need to be proposing changes I can just hit accept or reject on (this is how both Claude Code and Cursor work btw - you don't have to allow it to write to any file you don't want or execute any command you don't want).
aiiizzz 1y ago
The fact that you use Claude, not gpt5 which is light-years ahead for coding, tells me all I need to know.
- elashri 1y ago
  I don't think it is true that GPT-5 is much better than Claude 4.1. And more important is that light year is distance measure and I am sure openAI data centers is still on earth.

by Rochus 1y ago

oliwary 1y ago

Hey! I would encourage you to try our Claude code instead, which is also part of your subscription. It's a CLI that takes care of many of the issues you encountered, as it works directly on the code files in a directory. No more copy pasting or unscrambling results. Likewise, it can run commands itself to e.g. compile or even test code.

Rochus 1y ago
I'm working on old hardware and not-recent Linux and compiler versions, and I have no confidence yet in allowing AI direct (write) access to my repositories. Instead I provided Claude with the source code of a transpiler to C (one file) which is known to work, uses the same IR as the new code generators were supposed to use. This is a controlled experiment with a clear and complete input and clear expectations and specifications of the output. I don't think I would be able to clearly isolate the contributions and assess the performance of Claude when it has access to arbitrary parts of the source code.
- kelnos 1y ago
  > I have no confidence yet in allowing AI direct (write) access to my repositories. You don't need to give it write access to your repositories, just to a source tree.
- torben-friis 1y ago
  On Claude you specifically accept any attempt to use a terminal command (optionally whitelisting) so there’s no risk that it will push force something or whatever. You can also whitelist with granularity, for example to enable it to use git to view git logs but not commit. You can just let it work, see what’s uncommitted after it’s over, and get rid of the result if you don’t like it.
- stavros 1y ago
  I use Claude Code with the Max plan, and the experience isn't far off from what you describe. You still need to understand the system and review the implementation, because it makes many mistakes. That's not the part it saves me time in, it saves me time in looking up the documentation. Other than that, it might be slower, because the larger the code change is, the more time I need to spend reviewing, and past a point I just can't be bothered. The best way I've found is to have it write small functions, and then I tell it to compose them together. That way, I know exactly what's happening in the code, and I can trust that it works correctly. Cursor is probably a better way to do that than Claude Code, though.

kelnos 1y ago

I hate to play the "you're holding it wrong" card, but when I started, I had more or less the same experience. Eventually you start to learn how to better talk to it in order to get better results. Something I've found useful with Claude Code is that it works a lot better if I give it many small tasks to perform to eventually get the big thing done, rather than just dumping the big thing in its lap. You can do this interactively (prompt, output, prompt, output, prompt, output...) or by writing a big markdown file with the steps to build it laid out.

Rochus 1y ago
I just tried to use it for something I consider it to provide the most benefit (for my case). Being able to fully delegate a complicated (and boring) part to a machine would give me more time for the things I'm really interested in. I think we are on the right track in this regard, but we still have a long way to go.
chillingeffect 1y ago
Similar here. AI works much better as a consultant than as a developer. I ask it all kinds of things I have suspicions and intuitions about and it provides clarity and examples. It's great for subroutines. Trying to make full programs is just too large of a space. It's difficult to communicate all the implicit requirements.
- jennyholzer 1y ago
  People who consistently consult LLMs for product direction or software feature design overwhelmingly appear to me as willfully ignorant dullards. I mean it's even further than willful ignorance. It's delight in one's own ignorance.
JeanMarcS 1y ago
This. For me (senior as in I am in the field since last century), that's how I use it. "I want to do that, with this data, to obtain that" I still do the part of my job that I got experience on, analyze the need, and use the AI like an assistant to do small libraries or part of code. Like these, errors have less chance to appear. Then I glue that together. For me the time ratio is best use like that. If I have to describe the whole thing, I'm not far from doing it myself, so there's no need for me. Important: I work alone, not in a team, so maybe it has an impact on my thought

epolanski 1y ago

Imho your post summarizes 90% of the posts I see about AI coding on HN: not understanding the tools, not understanding their strenghts and weaknesses, not being good at prompting or context management yet forming strong(ish) opinions. If you don't know what they are good at and how to use them of course you may end up with mixed results and yes, you may waste time. That's a criticism I have also towards AI super enthusiasts (especially vibe coders, albeit you won't find much here), they often confuse the fact that LLMs often one shot 80% of the solutions with the idea that LLMs are 80% there, whereas the Pareto principle well applies to software development where it's the hardest 20% that's gonna prove difficult.

Rochus 1y ago
I'm pretty good at prompting and I successfully use Perplexity (mostly with Claude Sonnet 4) to develop concepts, sometimes with the same session expanded over several days. I think the user interface is much superior over Claude.ai. My hope was that the newer Claude Opus 4.1 would be much better in solving complicated coding tasks, which doesn't seem to be the case. For this I had to subscribe to claude.ai. Actually I didn't see much difference in performance, but a much worse UI and availability experience. When it comes to developing a complex topic in a factual dialogue, Claude Sonnet Thinking seems to me to be even more suitable than Claude Opus.
- epolanski 1y ago
  I'll be more detailed in my second reply. 1) Your original post asks a lot if not too much out the LLM, the expectation you have is too big, to the point that to get anywhere near decent results you need a super detailed prompt (if not several spec documents) and your conclusion stands true: it might be faster to just do it manually. That's the state of LLMs as of today. Your post neither hints at such detailed and laborious prompting nor seem to recognize you've asked it too much, displaying that you are not very comfortable with the limitations of the tool. You're still exploring what it can and what it can't do. But that also implies you're yet not an expert. 2) The second takeaway that you're not yet as comfortable with the tools as you think you are is clearly context management. 2/3k locs of code are way too much. It's a massive amount of output to hope for good results (this also ties with the quality of the prompt, with the guidelines and code practices provided, etc, etc). 3) Neither 1 or 2 are criticisms of your conclusions or opinions, if anything, they are confirmations of your point that LLMs are not there. But what I disagree with is the rush into concluding that AI coding provides net 0 benefits out of your experience. That I don't share. Instead of settling on what it could do (help with planning, writing a spec file, writing unit tests, providing the more boilerplate-y part of the code) and use the LLM to reduce the friction (and thus provide a net benefit), you essentially asked it to replace you and found out the obvious: that LLMs cannot take care of non-trivial business logic yet, and even when they can the results are nowhere near being satisfactory. But that doesn't mean that AI-assisted coding is useless and the net benefit is 0, or negative, it only becomes so as the expectations on the tool are too big and the amount of information provided is either too small to return consistent results or too large for the context to be an issue.

furyofantares 1y ago

I'm just shy of 30 years experience. I think I've spent more time learning how to use these tools than any other technology I've learned, and I still don't know the best way to use them. They certainly weren't a time-saver right away but they became one after some time giving them a real shot. I tested their + my limits on small projects, working out how to get them to do the whole project, figuring out when they stop working and why, figuring out which technology they work best with, figuring out the right size problems to give them, figuring out how to recognize if I'm asking them something they can't do well a ask something different instead, guide them into creating code that they can't actually continue to be successful with. I started last December in Cursor's agentic mode and have been in Claude Code ever since probably March or April. It's definitely been a huge boost all year for side projects - but only in the last couple months have I been having success in a large codebase. Even with all this experience I don't know that I would really be able to get much value out of the chat interface for them. They need to be proposing changes I can just hit accept or reject on (this is how both Claude Code and Cursor work btw - you don't have to allow it to write to any file you don't want or execute any command you don't want).

aiiizzz 1y ago

The fact that you use Claude, not gpt5 which is light-years ahead for coding, tells me all I need to know.

elashri 1y ago
I don't think it is true that GPT-5 is much better than Claude 4.1. And more important is that light year is distance measure and I am sure openAI data centers is still on earth.