4 ms·

I think the article misses the point. Yes you can find countless examples of a tool being used badly or even a tool being used correctly and providing bad resul

I think the article misses the point. Yes you can find countless examples of a tool being used badly or even a tool being used correctly and providing bad results. But while people spend time looking for flaws in a tool, (in what feels to me a bit as thinly disguised denialism of the inevitable) I prefer to ship features faster to my customers by using the tools to boost my productivity, not finding anecdotes that feel good where I caught the tool doing something stupid.

selcuka 1y ago
Exactly. I don't think LLMs are fully there yet, but that shouldn't stop us from taking advantage of the limited things they already do well.
imiric 1y ago
> while people spend time looking for flaws in a tool [...] I prefer to ship features faster to my customers by using the tools to boost my productivity I'm curious: do you review the AI generated code? In your experience, what is the percentage of times you've found it to be correct and shippable? I.e. not just that it compiles and does what you asked it to do, but that it doesn't contain security or performance issues, that it doesn't include useless or dead code, that is idiomatic, etc. Do you add tests for that code, written by yourself or the AI, and if so, how exhaustive are they?
- eranation 1y ago
  Yes of course, and the fact you have to ask these days (I would have asked the same) show how bad the situation is. I review a lot, revert a lot, reevaluate my life choices at times, it’s like supervising a toddler with 20 years of experience. It definitely contains security issues at times, perhaps less obvious ones, but subtle and harder to detect ones (the issues move up the stack). It’s almost never correct and shippable on the first go, but it still saves me and my team a lot of time on boilerplate and heavy lifting, and let us focus on creating value for customers. It’s still software engineering, just the actual coding part is much more productive (Coding has never been the main activity that software engineers spend most of their time on. They design, gather requirements, define scope, architect, plan, test, review, gather feedback, iterate, etc. Now they simply have more time to do these activities)
  - imiric 1y ago
    We've had similar experiences then, but I've arrived at a different conclusion. For me having to constantly babysit it, review and debug its output, and go through many rounds of back and forth without getting not just quality code, but working code at all, takes more time and effort than if I had done the research and written the code myself. You're right that actually typing code is not our main activity. It's the last part of the process. To get there we need to understand the domain and the problem, do some research and come up with a design and implementation plan, and maybe do some exploratory work. LLMs can certainly help with writing boilerplate code, and it's what I've found them most useful for as well. They're also useful as slightly better rubber ducks, to get ideas I might not have thought about, or to assist with design, architecture, testing, etc. All this is fine, if we ignore the hallucination issues. My problem is when I start asking them to write anything more sophisticated. Even when I give them a fairly detailed implementation plan (that they've helped me write), or point out an issue in great detail, more often than not they will lead me down wrong rabbit holes, produce non-working code I have to fix, and just waste my time. There have been very few cases where they've actually suggested a fix that turned out to be correct, or where what they've produced is usable, even after many iterations. > Now they simply have more time to do these activities That's the thing. I still have to do all those things you mentioned, while also babysitting an extremely confident junior engineer. This is why I feel that these tools are making me _less_ productive overall.

selcuka 1y ago

Exactly. I don't think LLMs are fully there yet, but that shouldn't stop us from taking advantage of the limited things they already do well.

imiric 1y ago

> while people spend time looking for flaws in a tool [...] I prefer to ship features faster to my customers by using the tools to boost my productivity I'm curious: do you review the AI generated code? In your experience, what is the percentage of times you've found it to be correct and shippable? I.e. not just that it compiles and does what you asked it to do, but that it doesn't contain security or performance issues, that it doesn't include useless or dead code, that is idiomatic, etc. Do you add tests for that code, written by yourself or the AI, and if so, how exhaustive are they?

eranation 1y ago
Yes of course, and the fact you have to ask these days (I would have asked the same) show how bad the situation is. I review a lot, revert a lot, reevaluate my life choices at times, it’s like supervising a toddler with 20 years of experience. It definitely contains security issues at times, perhaps less obvious ones, but subtle and harder to detect ones (the issues move up the stack). It’s almost never correct and shippable on the first go, but it still saves me and my team a lot of time on boilerplate and heavy lifting, and let us focus on creating value for customers. It’s still software engineering, just the actual coding part is much more productive (Coding has never been the main activity that software engineers spend most of their time on. They design, gather requirements, define scope, architect, plan, test, review, gather feedback, iterate, etc. Now they simply have more time to do these activities)
- imiric 1y ago
  We've had similar experiences then, but I've arrived at a different conclusion. For me having to constantly babysit it, review and debug its output, and go through many rounds of back and forth without getting not just quality code, but working code at all, takes more time and effort than if I had done the research and written the code myself. You're right that actually typing code is not our main activity. It's the last part of the process. To get there we need to understand the domain and the problem, do some research and come up with a design and implementation plan, and maybe do some exploratory work. LLMs can certainly help with writing boilerplate code, and it's what I've found them most useful for as well. They're also useful as slightly better rubber ducks, to get ideas I might not have thought about, or to assist with design, architecture, testing, etc. All this is fine, if we ignore the hallucination issues. My problem is when I start asking them to write anything more sophisticated. Even when I give them a fairly detailed implementation plan (that they've helped me write), or point out an issue in great detail, more often than not they will lead me down wrong rabbit holes, produce non-working code I have to fix, and just waste my time. There have been very few cases where they've actually suggested a fix that turned out to be correct, or where what they've produced is usable, even after many iterations. > Now they simply have more time to do these activities That's the thing. I still have to do all those things you mentioned, while also babysitting an extremely confident junior engineer. This is why I feel that these tools are making me _less_ productive overall.