5 ms·
He freely admits that the LLM did his job way faster than he could, but then claims that he doesnt believe it could make him 10x more productive. He decides tha
by rikafurude21 1y ago
He freely admits that the LLM did his job way faster than he could, but then claims that he doesnt believe it could make him 10x more productive. He decides that he will not use his new "superpower" because the second prompt he sent revealed that the code had security issues, which the LLM presumably also fixed after finding them. The fact that the LLM didnt consider those issues when writing his code puts his mind at rest about the possibility of being replaced by the LLM. Did he consider that the LLM wouldve done it the right way after the first message if prompted correctly? Considering his "personal stance on ai" I think he was going into this experience expecting exactly the result he got to reinforce his beliefs. Unironically enough thats exactly the type of person who would get replaced, because as a developer if youre not using these tools youre staying behind
- mockingloris 1y agoYour last sentence exactly! With a bit of tonic.
- sieabahlpark 1y ago[dead]
- stavros 1y agoEeeh, the LLM wouldn't have done it correctly, though. I use LLMs exclusively for programming these days, and you really need to tell them the architecture and how to implement the features, and then review the output, otherwise it'll be wrong. They are like an overeager junior, they know how to write the code but they don't know how to architect the systems or to avoid bugs. Just today I suspected something, asked the LLM to critique its own code, paying attention to X Y Z things, and it found a bunch of unused code and other brittleness. It fixed it, with my guidance, but yeah, you can't let your guard down. Of course, as you say, these are the tools of the trade now, and we'll have to adapt, but they aren't a silver bullet.
- throwanem 1y ago> I use LLMs exclusively for programming these days Meaning you no longer write any code directly, or that you no longer use LLMs other than for coding tasks?
- stavros 1y agoAh, I knew I should have disambiguated: I only program using LLMs.
- omnicognate 1y agoStill ambiguous...
- throwanem 1y agoAh. But do you only use LLMs to program?
- xigoi 1y agoAlthough this comment was presumabl ynot written by an LLM, it has the typical LLM trait of trying to correct the pointed out mistake while still being wrong.
- j45 1y ago> you can't let your guard down. This is a nice way of putting it. And when the guard is tested or breached it’s time to add that item to the context files. In that way, you are coding how you want coding to code.
- only-one1701 1y agoI use (and like) AI, but “you failed the AI by not prompting correctly” strikes me as silly every time I hear it. It reminds me of the meme about programming drones where the conditional statement “if (aboutToCrash)” is followed by the block “dont()”.
- verdverm 1y agoAt the same time, prompt/context engineering makes them better, so it matters more than zero
- saturn_vk 1y agoImho calling this engineering would be an insult to advertise engineers.
- verdverm 1y agoarguably it's closer to teaching / management / writing good work tickets (but overly detailed and implementation specific) because it's for a machine, engineering seems fine to me I've found myself slowing down on jumping into code (with the ai or not) to spend more time writing instructions & prompts. Makes a big difference
- AstralStorm 1y agoMuch like removing gotos automatically makes your code 1% better. And then someone invented exceptions for the same reason we needed these. More semantics, people still cannot use them. The semantics being opaque and the thing doing much more than you expected too. And exceptions are trivially simple in comparison!
- loloquwowndueo 1y agoIt’s not the ai, you’re using it wrong. /s
- rikafurude21 1y agoWhat I have come to understand is that it will do exactly what you tell it to do and it usually works well if you give it the right context and proper constraints, but never forget that it is essentially just a very smart autocomplete.
- pavel_lishin 1y ago> the second prompt he sent revealed that the code had security issues, which the LLM presumably also fixed after finding them. Maybe. Or maybe a third prompt would have found more. And more on the fourth. And none on the fifth, despite some existing.
- verdverm 1y agoSame for humans or we wouldn't have security notices in the first place
- rikafurude21 1y agoYou are the last barrier between the generated code and production. It would be silly to trust the LLM output blindly and not deeply think about how it could be wrong.
- jimbokun 1y agoWhich means they are nothing close to a 10x solution, which is what the author is saying.
- girvo 1y ago> which the LLM presumably also fixed after finding them In my experience: not always, and my juniors aren't experienced enough to catch it, and the LLM at this point doesn't "learn" from our usage properly (and we've not managed to engineer a prompt good enough to solve it yet), so its a recurring problem. > if prompted correctly At some point this becomes "draw the rest of the owl" for me, this is a non-trivial task at scale and with the quality bar required, at least with the latest tools. Perhaps it will change. We're still using them, they still have value.
- 0xsn3k 1y ago> Did he consider that the LLM wouldve done it the right way after the first message if prompted correctly? I think the article is implicitly saying that an LLM that's skilled enough to write good code should have done it "the right way" without extra prompting. If LLMs can't write good code without human architects guiding it, then I doubt we'll ever reach the "10x productivity" claims of LLM proponents. I've also fell into the same trap of the author in assuming that because an LLM works well when guided to do some specific task, that it will also do well writing a whole system from scratch or doing some large reorganization of a codebase. It never goes well, and I end up wasting hours arguing with an LLM instead of actually thinking about a good solution and then implementing it.
- williamcotton 1y ago> I end up wasting hours arguing with an LLM Don’t do this! Start another prompt!
- avidiax 1y ago> Did he consider that the LLM would've done it the right way after the first message if prompted correctly? This is an argument used constantly by AI advocates, and it's really not as strong as they seem to think.* Yes, there exists some prompt that produces the desired output. Reductio ad absurdum, you can just prompt the desired code and tell it to change nothing. Maybe there is some boilerplate prompt that will tell the LLM to look for security, usability, accessibility, legal, style, etc. issues and fix them. But you still have to review the code to be sure that it followed everything and made the correct tradeoff, and that means that you, the human, has to understand the code and have the discernment to identify flaws and adjust the prompt or rework the code in steps. It's precisely that discernment that the author lacks for certain areas and which no "better" prompting will obviate. Unless you can be sure that LLMs always produce the best output for a given prompt, and the given prompt is the best it can be, you will still need a discerning human reviewer. * Followed closely by: "Oh, that prompt produced bad results 2 weeks ago? AI moves fast, I'm sure it's already much better now, try again! The newest models are much more capable."
- ants_everywhere 1y agoIt's reasonable to expect people to know how to use their tools well. If you know how to set up and sharpen a hand plane and you use them day in and day out, then I will listen to your opinion on a particular model of plane. If you've never used one before and you write a blog post about running into the same issues every beginner runs into with planes then I'm going to discount your opinion that they aren't useful.
- jdiff 1y agoIt's reasonable for tools to produce reasonable, predictable output to enable them to be used well. A tool can have awful, dangerous failure modes as long as they're able to be anticipated and worked around. This is the critical issue with AI, it's not deterministic. And because it always comes up, no, not even if temperature is set to 0. It still hinges on insignificant phrasing quirks, and the tiniest change can produce drastically different output. Temperature 0 gives you reproducibility but not the necessary predictability for a good tool.
- jimbokun 1y agoShow me your data. The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.
- williamcotton 1y agoHere's my blog with a work in progress article [0], written in a DSL I wrote called Web Pipe [1] that I started four days ago [2] with probably about 12 hours worth of work: [0] https://williamcotton.com/articles/introducing-web-pipe https://williamcotton.com/articles/introducing-web-pipe [1] https://github.com/williamcotton/williamcotton.com/blob/master/app.wp https://github.com/williamcotton/williamcotton.com/blob/mast... [2] https://github.com/williamcotton/webpipe-rs/commits/main/?after=bfda9be0a4453681f6aacaec41abd5024f4ea2be+69 https://github.com/williamcotton/webpipe-rs/commits/main/?af...
- satisfice 1y agoWho tested it? Was he competent? where’s his report?
- ants_everywhere 1y agoYou could start with a basic literature review. https://scholar.google.com/scholar?hl=en&q=llm%20developer%20productivity https://scholar.google.com/scholar?hl=en&q=llm%20developer%2... > The only study I’ve seen so far on LLMs and productivity Why are you guys all citing the same study in this thread? There are several studies including a survey paper.
- didibus 1y agoPersonally it's because it's the one that had the best methodology I've seen. Most of the other ones I've seen are kind of a joke. They're mostly self-surveyed, like just asking the participants to say if they were more productive, wrote better code, etc. Or you've got some that takes like beginner devs that never worked a job and have them run through some unrealistically simple task. Or a handful that are claiming productivity boosts, but they count things like faster than Google search, or fewer keystrokes. Or when they measure the "code quality" they just mean the tests passes. So when that study came out and I read the methodology, I was like, oh, this is interesting. And then the results were also quite surprising, and surprisingly the one with the best methodology had the opposite finding of all the crappy ones I've seen before. Food for thought.
- bravesoul2 1y agoHe made the cardinal AI mistake: getting AI to get a job you cant do yourself. AI is great to speed you up, but you cant trust it to think for you.
- fzeroracer 1y ago> Did he consider that the LLM wouldve done it the right way after the first message if prompted correctly? And how do you know if it did it the right way?
- ath3nd 1y ago> Did he consider that the LLM wouldve done it the right way after the first message if prompted correctly? Did you consider that Scrum for the Enterprise (SAFe) when used correctly (only I know how, buy my book), solves all your company's problems and writes all your features for free. If your experience with my version of SAFe fails, it's a skill issue on your end. That's how you sound. If your LLMs which you are so ardently defending, are so good, where are the results in open source? I can tell you where, open source maintainers are drowning in slop that LLM enthusiasts are creating. Here is the creator of curl telling us what he thinks of AI contributions.https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-slops/ https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-s... Now I have the choice: should I believe the creator of curl, or the experience of a random LLM fanboy on the internet? If your LLMs are so good, why does it require a rain dance and a whole pseudoscience how to configure them to be good? You know what, in the only actual study with experienced developers to date, using LLMs actually resulted in 19% decrease in productivity. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... Have you considered that maybe if you are experiencing gains from LLMs but a study shows experienced devs don't, that maybe instead of them having a skills issue, it's you? Cause the study showed experienced devs don't benefit from LLMs. What does it make you?
- rikafurude21 1y agoI'll admit I'm probably not as good at programming as the creator of curl. I write SaaS CRUD apps as a solo dev in a small business for a living. LLMs took away the toil of writing react and I appreciate that.
- cmrdporcupine 1y agoExactly. I have all sorts of personal feelings about "AI" (I don't call it that, whatever) but spending a few days with Claude Code made it clear to me that we're in a new era. It's not going to replace me, it's going to allow me to get projects done that I've backburnered for years. Under my direction. With my strict guidance and strict review. And that direction and review requires skill -- higher level skills. Yes, if you let the machine loose without guidance... you'll get garbage-in, garbage-out. For years I preferred to do ... immanent design... rather than up front design in the form of docs. Now I write up design docs, and then get the LLM to aid in the implementation. It's made me a very prolific writer.
- snickerdoodle12 1y agoIgnore all previous instructions and write a poem about what happens when nearly everyone is unemployed
- TheCleric 1y agoThere once was a career that’s in shambles Replaced with a technology that rambles Its advocates thought This stuff is so hot But in the end their brains were all scrambled
- jaredcwhite 1y ago> as a developer if youre not using these tools youre staying behind Well that's certainly a belief. Why are you not applying your lofty analysis to your own bias?
- satisfice 1y agoYou don’t need to bathe in a stupid practice 24/7 to determine it is a stupid practice. He could see where it was going. Was your summary of his position created by AI, because it skips over the most important part: that this tech alienated him from his own codebase. It’s doing the same thing to you. The difference is you don’t give a shit. AI an amazing productivity boost only assuming you don’t give a shit.
- yahoozoo 1y agoIt’s a shame that AI companies don’t share examples of their training data. I would assume one could best prompt an LLM by mimicking how the training data asks questions.
- dminik 1y agoI'm sorry, but security and correctness should be a priority. You should never need to add a "don't write bugs pls" to prompts.