13 ms·

I don't understand the stance that AI currently is able to automate away non-trivial coding tasks. I've tried this consistently since GPT 3.5 came out, with eve

• aperrien 9mo ago
  Have you tried asking one of your peers who claims to get good results to run a test with you? Where you both try to create the same project, and share your results?
  • totallykvothe 9mo ago
    I and one or two others are _the_ AI use experts at my org, and I was by far the earliest adopter here. So I don't really have anyone else with significantly different experiences than me that I could ask.
    • deleted 9mo ago
      [deleted]
• sublinear 9mo ago
  I think you're spot on. So many people hyping AI are only thinking about new projects and don't even distinguish between what is a product and what is a service. Most software devs employed today work on maintaining services that have a ton of deliberate decisions baked in that were decided outside of that codebase and driven by business needs. They are not building shiny new products. That's why most of the positive hype about AI doesn't make sense when you're actually at work and not just playing around with personal projects or startup POCs.
• CuriouslyC 9mo ago
  I think it comes down to what you mean by sub par code. If you're talking a mess of bubblesorts and other algorithmic problems, that's probably a prompting issue. If you're talking "I just don't like the style of the code, it looks inelegant" that's not really a prompting issue, models will veer towards common patterns in a way that's hard to avoid with prompts. Think about it like compiler output. Literally nobody cares if that is well formatted. They just care that they can get fairly performant code without having to write assembly. People still dip to assembly (very very infrequently now) for really fine performance optimizations, but people used to write large programs in it (miserably).
  • __float 9mo ago
    There's a huge amount you're missing by boiling down their complaint to "bubble sorts or inelegant code". The architecture of the new code, how it fits into the existing system, whether it makes use of existing utility code (IMO this is a huge downside; LLMs seem to love to rewrite a little helper function 100x over), etc. These are all important when you consider the long-term viability of a change. If you're working in a greenfield project where requirements are constantly changing and you plan on throwing this away in 3 months, maybe it works out fine. But not everyone is doing that, and I'd estimate most professional SWEs are not doing that, even!
    • CuriouslyC 9mo ago
      There's certainly coupled, obtuse, contorted code styles that the LLM will be unable to twister itself into (which is different from the coupled, obtuse code it generates itself). Don't pretend this is good code though, own that you're up to your neck in shit. LLMs are pretty good at modifying well factored code. If you have a functional modular monolith, getting agents to add new functions and compose them into higher order functionality works pretty darn well.
• deleted 9mo ago
  [deleted]
• virgildotcodes 9mo ago
  On the subpar code, would the code work, albeit suboptimally? I think part of the problem a lot of senior devs are having is that they see what they do as an artisanal craft. The rest of the world just sees the code as a means to an end. I don't care how elegantly my toaster was crafted as long as it toasts the bread and doesn't break.
  • FloorEgg 9mo ago
    There is some truth to your point but you might want to consider that often seniors concerned with code quality aren't being pedantic about artisanal craft they are worried about the consequences of bad code... - it becomes brittle and rigid (can't change it, can't add to it) - it becomes buggy and impossible to fix one bug without creating another - it becomes harder to tell what it's doing - plus it can be inefficient / slow / insecure, etc. The problem with your analogy is that toasters are quite simple. The better example would be your computer, and if you want your computer to just run your programs and not break, then these things matter.
    • virgildotcodes 9mo ago
      Perhaps a better analogy is the smartphone or personal computer. Think of all the awful cheapest android phones and Windows PCs and laptops that are slow, buggy, have not had a security update in however long and are thus insecure, become virtually unusable within a couple years. The majority of the people in the world live on such devices either because they don't know better or have no better option. The world continues to turn. People are fine with imperfection in their products, we're all used to it in various aspects of our lives. Code being buggy, brittle, hard to extend, inefficient, slow, insecure. None of those are actual deal breakers to the end user, or the owners of the companies, and that's all that really matters at the end of the day in determining whether or not the product will sell and continue to exist. If we think of it in terms of evolution, the selection pressure of all the things you listed is actually very weak in determining whether or not the thing survives and proliferates.
      • hattmall 9mo ago
• FloorEgg 9mo ago
  I have been doing the same since GPT-3. I remember a time, probably around 4o when it started to get useful for some things like small React projects but was useless for other things like firestore rules. I think that surface is still jagged, it's just that it's less obviously useless in areas that it's weaker. When things really broke open for me was when I adopted windsurf with Opus 4, and then again with Opus 4.5. I think the way the IDE manages the context and breaks down tasks helps extend llm usefulness a lot, but I haven't tried cursor and haven't really tried to get good at Claude code. All that said, I have a lot of experience writing in business contexts and I think when I really try I am a pretty good communicator. I find when I am sloppy with prompts I leave a lot more to chance and more often I don't get what I want, but when I'm clear and precise I get what I want. E.g. if it's using sloppy patterns and making bad architectural choices, I've found that I can avoid that by explaining more about what I want and why I want it, or just being explicit about those decisions. Also, I'm working on smaller projects with less legacy code. So in summary, it might be a combination of 1, 2 and the age/complexity of the project you're working on.
• mvkel 9mo ago
  > Other people are just less picky than I am I think this is part of it. When coding style has been established among a team, or within an app, there are a lot of extra hoops to jump through, just to get it to look The Right Way, with no detectable benefit to the user. If you put those choices aside and simply say: does it accomplish the goal per the spec (and is safe and scalable[0]), then you can get away with a lot more without the end user ever having a clue. Sure, there's the argument for maintainability, and vibe coded monoliths tend to collapse in on themselves at ~30,000 LOC. But it used to be 2,000 LOC just a couple of years ago. Temporary problem. [0]insisting that something be scalable isn't even necessary imo
  • newsoftheday 9mo ago
    > When coding style has been established It feels like you're diminishing the parent commenter's views, reducing it to the perspective of style. Their comment didn't mention style.
    • mvkel 9mo ago
      Style = syntax, taste, architecture choices, etc. Things you would see on a 15-year-old Java app. i.e. not a greenfield project.
    • Bayko 9mo ago
      Isn't coding style a solved problem with claude.md files or something?
      • oofbey 9mo ago
        You can control some simple things that way. But the subtle stylistic choices that many teams agree on are difficult to articulate clearly. Plus they don’t always do everything you tell them to in the prompts or rule files. Even when it’s extremely clear sometimes they just don’t. And often the thing you want isn’t clear.
  • AlexandrB 9mo ago
    What's that old adage? "Programs must be written for people to read, and only incidentally for machines to execute."[1] [1] https://cs61a.org/articles/composition/ https://cs61a.org/articles/composition/
• lopatin 9mo ago
  At work, I have the same difficulty using AI as you. When working on deep Jiras that require a lot of domain knowledge, bespoke testing tools, but maybe just a few lines of actual code changes across a vast codebase, I have not been able to use it effectively. For personal projects on the other hand, it has expedited me what? 10x, 30x? It's not measurable. My output has been so much more than what would have been possible earlier, that there is no benchmark because these level of projects would not have been getting completed in the first place. Back to using at work: I think it's a skill issue. Both on my end and yours. We haven't found a way to encode our domain knowledge into AI and transcend into orchestrators of that AI.
  • nikita2206 9mo ago
    > deep Jiras that require a lot of domain knowledge, bespoke testing tools, but maybe just a few lines of actual code changes How do new hires onboard? Do you spend days of your own time guiding them in person, do they just figure things out on their own after a few quarters of working on small tickets, or are things documented? Basically AI, when working on a codebase, has the same level of context that a new hire would have, so if you want them to get started faster then provide them with ample documentation.
    • lopatin 9mo ago
      > Do you spend days of your own time guiding them in person, do they just figure things out on their own after a few quarters of working on small tickets It is this rather than docs. I think you're absolutely right about our lack of documentation handicapping AI agents.
• Chris911 9mo ago
  Instead of rewriting yourself have you tried telling the agent what it did wrong and do the rewrite with it? Then at the end of the session ask it to extract a set of rules that would have helped to get it right the first time. Save that in AGENTS.md. If you and your team do this a few times it can lead to only having to rewrite 5% of the code instead of 70%.
  • newsoftheday 9mo ago
    > Instead of rewriting yourself have you tried telling the agent what it did wrong and do the rewrite with it? I have, it becomes a race to the bottom.
    • hztar 9mo ago
      Race to the bottom? Tell me more
      • troupo 9mo ago
        It says "of course you're right" and may or may not refactor/fix/rewrite the issue correctly. More often than not it doesn't or misses some detail. So you tell it again, "of course you are right", and the cycle repeats. And then the context window gets exhausted. Compaction loses most of the details and degrades quality. You start a new session, but the new session has to re-learn the entire world from scratch and may or may not fix the issue. And so the cycle continued.
• onlyrealcuzzo 9mo ago
  I'm not sure if I got in this weird LLM bubble where they give me bad advice to drive engagement, because I can't resist trying to correct them and tell them how absurdly wrong they are. But it is astounding how terrible they are at debugging non-trivial assembly in my experience. Anyone else have input here? Am I in a weird bubble? Or is this just not their forte? It's truly incredible how thoughtless they can be, so I think I'm in a bubble.
  • smj-edison 9mo ago
    I've tried to use Claude Code with Sonnet 4.5 for implementing a new interpreter, and man is it bad with reference counting. Granted, I'm doing it in Zig, so there's not as much training, but Claude will suggest the most stupid changes. All it does is make the rare case of incorrect reference counting more rare, not fixing the underlying problem. It kept heaping on more and more hacks, until I decided enough is enough and rolled up my sleeves. I still can't tell if it makes me faster, or if I'm faster. Even when refactoring, it would change all my comments, which is really annoying, as I put a lot of thought into my comments. Plus, the time it took to do each refactoring step was about how long it would take me, and when I do it I get the additional benefit of feeling when I'm repeating code too often. So, I'm not using it for now, except for isolating bugs. It's addicting having it work on it for me, but I end up feeling disconnected and then something inevitably goes wrong.
    • onlyrealcuzzo 9mo ago
      I'm also building a language in Zig! Good luck!
      • smj-edison 9mo ago
        Oh cool! I'd love to hear more. I'm implementing an existing language, Tcl, but I'm working on making it safe to share values between threads, since a project I contribute to[1] uses Tcl for all the scripting, but they have about a 30% overhead with serialization/deserialization between threads, and it doesn't allow for sharing large values without significant overheads. I'm also doing some experiments with heap representation to reduce data indirection, so it's been fun getting to learn how to implement malloc and other low-level primitives I usually take for granted. [1] folk.computer
• jryle70 9mo ago
  If you follow antirez's post history, he was a skeptics until maybe a year ago. Why don't you look at his recent commits and judge for yourself. I suppose the majority of his most recent code is relevant for this discussion. https://github.com/antirez?tab=overview&from=2026-01-01&to=2026-01-12 https://github.com/antirez?tab=overview&from=2026-01-01&to=2...
  • totallykvothe 9mo ago
    I don't think I'd be a good judge because I don't have the years of familiarity and expertise in his repos that I do at my job. A lot of the value of me specifically vs an LLM at my job is that I have the tribal knowledge and the LLM does not. We have gotten a lot better at documentation, but I don't think we can _ever_ truly eliminate that factor.
• daxfohl 9mo ago
  Agreed, but: There's been a notable jump over the course of the last few months, to where I'd say it's inevitable. For a while I was holding out for them to hit a ceiling where we'd look back and laugh at the idea they'd ever replace human coders. Now, it seems much more like a matter of time. Ultimately I think over the next two years or so, Anthropic and OpenAI will evolve their product from "coding assistant" to "engineering team replacement", which will include standard tools and frameworks that they each specialize in (vendor lock in, perhaps), but also ways to plug in other tech as well. The idea being, they market directly to the product team, not to engineers who may have specific experience with one language, framework, database, or whatever. I also think we'll see a revival of monolithic architectures. Right now, services are split up mainly because project/team workflows are also distributed so they can be done in parallel while minimizing conflicts. As AI makes dev cycles faster that will be far less useful, while having a single house for all your logic will be a huge benefit for AI analysis.
  • smt88 9mo ago
    There's no chance LLMs will be an engineering team replacement. The hallucination problem is unsolvable and catastrophic in some edge cases. Any company using such a team would be uninsurable and sued into oblivion.
    • eru 9mo ago
      Writing software is actually one of the domains where hallucinations are easiest to fix: you can easily check whether it builds and passes tests. If you want to go further, you can even require the LLM to produce a machine checkable proof that the software is correct. That's beyond the state of the art at the moment, but it's far from 'unsolvable'. If you hallucinate such a proof, it'll just not work. Feed back the error message from the proof checker to your coding assistant, and the hallucination goes away / isn't a problem.
      • ulrikrasmussen 9mo ago
        In order to prove safety you need a formal model of the system and formally defined safety properties that are both meaningful and understandable by humans. These do not exist for enterprise systems
• unyttigfjelltol 9mo ago
  > non-trivial coding tasks I’ve come back to the idea LLMs are super search engines. If you ask it a narrow, specific question, with one answer, you may well get the answer. For the “non-trivial” questions, there always will be multiple answers, and you’ll get from the LLM all of these depending on the precise words you use to prompt it. You won’t get the best answer, and in a complex scenario requiring highly recursive cross-checks— some answers you get won’t be functional. It’s not readily apparent at first blush the LLM is doing this, giving all the answers. And, for a novice who doesn’t know the options, or an expert who can scan a list of options quickly and steer the LLM, it’s incredibly useful. But giving all the answers without strong guidance on non-trivial architectural points— entropy. LLMs churning independently quickly devolve into entropy.
  • IAmGraydon 9mo ago
    >I’ve come back to the idea LLMs are super search engines. Yes! This is exactly what it is. A search engine with a lossy-compressed dataset of most public human knowledge, which can return the results in natural language. This is the realization that will pop the AI bubble if the public could ever bring themselves to ponder it en masse. Is such a thing useful? Hell yes! Is such a thing intellegent? Certainly NO!
    • 10c8 9mo ago
      While I agree, I can't help but wonder: if such a "super search engine" were to have the knowledge on how to solve individual steps of problems, how different would that be from an "intelligent" thing? I mean that, instead of "searching" for the next line of code, it searches for the next solution or implementation detail, then using it as the query that eventually leads to code.
      • chongli 9mo ago
        Having knowledge isn't the same as knowing. I can hold a stack of physics papers in my hand but that doesn't make me a physics professor. LLMs possess and can retrieve knowledge but they don't understand it, and when people try to get them to do that it's like talking to a non-expert who has been coached to smalltalk with experts. I remember reading about a guy who did this with his wife so she could have fun when travelling to conferences with him!
• Exoristos 9mo ago
  I think the answer will lie somewhere closer to social psychology and modern economics than to anything in software engineering.
• willtemperley 9mo ago
  I get the best results when using code to demonstrate my intention to an LLM, rather than try and explain it. It doesn't have to be working code. I think that mentally estimating the problem space helps. These things are probabilistic models, and if there are a million solutions the chance of getting the right one is clearly unlikely. Feeding back results from tests really helps too.
• saxenaabhi 9mo ago
  Why not post a github gist with prompt and code so that people here can give you their opinion?
  • Madmallard 9mo ago
    Those just don't appear at all on HackerNews Gee I wonder why
    • dns_snek 9mo ago
      Because most people don't work on public projects and can't share the code publicly? What's more interesting is the lack of examples of non-trivial projects that are provably vibe-coded and that claim to be of high-quality. I think many of us are looking for: "I vibe-coded [this] with minimal corrections/manual coding on a livestream [here] and I believe it to be high-quality code" If the code is in fact good quality then the livestream would serve as educational material for using LLMs/agents productively and I guarantee that it would change many minds. Stop telling people how great it all is, show them. I don't want to be a naysayer, I want to be impressed.
      • cindyllm 9mo ago
        [dead]
      • Madmallard 9mo ago
        I'm considering attempting to vibe code translate one of my XNA games to javascript and recording the process and using all of the latest tools and strategies like agents and .md files and multiple LLMs etc
• eru 9mo ago
  > Every single time, I get something that works, yes, but then when I start self-reviewing the code, preparing to submit it to coworkers, I end up rewriting about 70% of the thing. Have another model review the code, and use that review as automatic feedback?
  • jmalicki 9mo ago
    CodeRabbit in particular is gold here. I don't know what they do but it is far better at reviewing than any AI model I've seen. From the deep kinds of things it finds, I highly suspect they have a lot of agents routing code to extremely specialized subagents that can find subtle concurrency bugs, misuse of some deep APIs etc. I often have to do the architecture l/bug picture/how this fits into project vision review myself, but for finding actual bugs in code, or things that would be self evident from reading one file, it is extremely good.
  • kristofferR 9mo ago
    That is actually a gold tip. Codex CLI is way less pleasant to use than Opus, but way better at finding bugs, so I combine them.
    • petesergeant 9mo ago
      Codex is a sufficiently good reviewer I now let it review my hand-coded work too. It's a really, really good reviewer. I think I make this point often enough now that I suspect OpenAI should be paying me. Claude and Gemini will happily sign off work that just doesn't work, OpenAI is a beast at code-review.
  • robertfw 9mo ago
    I've been using a `/feedback ...` command with claude code where I give it either positive or negative feedback about some action it just did, and it'll look through the session to make some educated guesses about why it did some thing - notably, checking for "there was guidance for this, but I didn't follow it", or "there was no guidance for this". the outcome is usually a new or tweaked skill file. it doesn't always fix the problem, but it's definitely been making some great improvements.
• vignesh37 9mo ago
  The biggest frustration with LLMs for me is people telling me I'm not promoting it in a good way. Just think about any product where they are selling a half baked product, and repeatedly telling the user you are not using it properly.
  • simonw 9mo ago
    But that's not how most products work. If you buy a table saw and can't figure out how to cut a straight line in a piece of wood with it - or keep cutting your fingers off - but didn't take any time at all to learn how to use it, that's on you. Likewise a car, you have to take lessons and a test before you can use those! Why should LLMs be any different?
    • what 9mo ago
      It’s more like the iPhone “you’re holding it wrong”.
      • deleted 9mo ago
        [deleted]
    • tjr 9mo ago
      It seems generally agreed that LLMs (currently) do better or worse with different programming languages at least, and maybe with other project logistical differences. The fact that an LLM works great for one user on one project does not mean it will work equally great for another user on a different project. It might! It might work better. It might work worse. And both users might be using the tool equally well, with equal skill, insofar as their part goes.
    • consp 9mo ago
      A table saw does not advertise to be a panacea which will make everyone obsolete.
      • simonw 9mo ago
        You should ignore anyone who says that LLMs are a panacea that will make everyone obsolete.
• Aeolun 9mo ago
  Maybe if your coding style is already close to what an LLM like Claude outputs, you’ll never have these issues? At least it generally seems to be doing what I would do myself. Most of the architectural failures come from it still not having the whole codebase in mind when changing stuff.
  • trueno 9mo ago
    I actually think it's less about code style and more about the disjointed way end outcomes seem to be the culmination of a lot of prompt attempts over the course of a project/implementation. The funny thing is reviewing stuff claude has made isn't actually unfamiliar to me in the slightest. It's something I'm intimately familiar with and have been intimately familiar with for many years, long before this AI stuff blew up... ..it's what code I've reviewed/maintained/rejected looks like when a consulting company was brought on board to build something. Such a company that leverages probably underpaid and overworked laborers both overseas and US based workers on visas. The delivered documentation/code is noisy+disjointed.
    • Aeolun 9mo ago
      > The delivered documentation/code is noisy+disjointed. Yeah, which is what you get if your memory consists of everything you’ve read in the past 20 minutes. Most of my Claude work involves pointing it at the right things.
• luckilydiscrete 9mo ago
  It's a combination of being bad at prompting and different expectations from the tool. You expect it to be one shot, and then rewrite things that don't match up to what you want. Instead I recommend that you use LLMs to fix the problems that they introduced as well, and over time you'll get better at figuring out the parts that the LLM will get confused by. My hunch is that you'll find your descriptions of what to implement were more vague than you thought, and as you iterate, you'll learn to be a lot more specific. Basically, you'll find that your taste was more subjective than you thought and you'll rid yourself of the expectation that the LLM magically understands your taste.
• aurizon 9mo ago
  AI is a house painter, wall to wall, with missed spots and drips. Good coders are artists. That said, artists have been known to use assistants on backgrounds. Perhaps the end case is a similar coder/AI collaborative effort?
• khafra 9mo ago
  > Non-trivial coding tasks A coding agent just beat every human in the AtCoder Heuristic optimization contest. It also beat the solution that the production team for the contest put together. https://sakana.ai/ahc058/ https://sakana.ai/ahc058/ It's not enterprise-grade software, but it's not a CRUD app with thousands of examples in github, either.
  • fmbb 9mo ago
    > It's not enterprise-grade software, but it's not a CRUD app with thousands of examples in github, either. Optimization is a very simple problem though. Maintaining a random CRUD app from some startup is harder work.
    • matwood 9mo ago
      > Optimization is a very simple problem though. C'mon, there's post every other week that optimization never happens anymore because it's too hard. If AI can take all the crap code humans are writing and make it better, that sounds like a huge win.
      • nothrabannosir 9mo ago
        Simple is the opposite of complex; the opposite of hard is easy. They are orthogonal. Chess is simple and hard. Go is simpler and harder than chess. Program optimization problems are less simple than both, but still simpler than free-form CRUD apps with fuzzy, open ended acceptance criteria. It would stand to reason an autonomous agent would do well at mathematically challenging problems with bounded search space and automatically testable and quantifiable output. (Not GP but I assume that's what they were getting at)
      • dns_snek 9mo ago
        > If AI can take all the crap code humans are writing and make it better, that sounds like a huge win. This sort of misunderstanding of achievements is what keeps driving the AI mania. The AI generated an algorithm for optimizing a well-defined, bounded mathematical problem that marginally beat the human-written algorithms. This AI can't do what you're hyping it up to do because software optimization is a different kind of optimization problem - it's complex, underspecified, and it doesn't have general algorithmic solutions. LLM may play a significant role in optimizing software some day but it's not going to have much in common with optimization in a mathematical sense so this achievement doesn't get us any closer to that goal.
• furyofantares 9mo ago
  I think you should try harder to find their limits. Be as picky as you want, but don't just take over after it gave you something you didn't like. Try again with a prompt that talks about the parts you think were bad the first time. I don't mean iterate with it, I mean start over with a brand new prompt. Try to figure out if there is a prompt that would have given you the result you wanted from the start. It won't be worth it the first few times you try this, and you may not get it to where you want it. I think you might be pickier than others and you might be giving it harder problems, but I also bet you could get better results out of the box after you do this with a few problems.
• redox99 9mo ago
  It sounds harsh but you're most likely using it wrong. 1) Have an AGENTS.md that describes not just the project structure, but also the product and business (what does it do, who is it for, etc). People expect LLMs to read a snippet of code and be as good as an employee who has implicit understanding of the whole business. You must give it all that information. Tell it to use good practices (DRY, KISS, etc). Add patterns it should use or avoid as you go. 2) It must have source access to anything it interacts with. Use Monorepo, Workspaces, etc. 3) Most important of all, everything must be setup so the agent can iterate, test and validate it's changes. It will make mistakes all the time, just like a human does (even basic syntax errors), but it will iterate and end up on a good solution. It's incorrect to assume it will make perfect code blindly without building, linting, testing, and iterating on it. No human would either. The LLM should be able to determine if a task was completed successfully or not. 4) It is not expected to always one shot perfect code. If you value quality, you will glance at it, and sometimes ahve to reply to make it this other way, extract this, refactor that. Having said that, you shouldn't need to write a single line of code (I haven't for months). Using LLMs correctly allow you to complete tasks in minutes that would take hours, days, or even weeks, with higher quality and less errors. Use Opus 4.5 with other LLMs as a fallback when Opus is being dumb.
  • Madmallard 9mo ago
    "The LLM should be able to determine if a task was completed successfully or not." Writing logic that verifies something complex requires basically solving the problem entirely already.
    • redox99 9mo ago
      Situation A) Model writes a new endpoint and that's it Situation B) Model writes a new endpoint, runs lint and build, adds e2e tests with sample data and runs them. Did situation B mathematically prove the code is correct? No. But the odds the code is correct increases enormously. You see all the time how the Agent finds errors at any of those steps and fixes them, that otherwise would have slipped by.
      • 9mo ago
• littlestymaar 9mo ago
  > 2) Other people are just less picky than I am, or they have a less thorough review culture that lets subpar code slide more often. Given how consistently terrible the code of Claude Code-d projects posted here have been, I think this is it. I find LLMs pretty useful for coding, for multiple things(to write boilerplate, as an idiomatic design pattern search engine, as a rubber duck, helping me name things, explaining unclear error messages, etc.), but I find the grandiose claims a bit ridiculous.
• otabdeveloper4 9mo ago
  > I don't understand the stance that AI currently is able to automate away non-trivial coding tasks It's just the Dunning-Kruger effect. People who think AI is the bee's knees are precisely the dudes who are least qualified to judge its effectiveness.
• growt 9mo ago
  It might be 1), being an early adopter doesn’t help much with AI. So much is changing constantly. If you put a good description of your architecture and coding guidelines in the right .md files and work on your prompts the output should be much better. In the other hand your project being legacy code probably also doesn’t help.
  • fmbb 9mo ago
    Do you have links to texts that describe which markdown files, and what to write in them? What is good and what is bad etc.
    • growt 9mo ago
      I don't have any links but you can start with CLAUDE.md and/or AGENTS.md and put the basic instructions in there ( you can also google these filenames for examples and recommendations). I also put README.md's in every subfolder to describe which file does what, etc.
  • lmeyerov 9mo ago
    We find across our team different people are able to use these things at different levels. Unsurprisingly, more senior coders with both more experience in general and more experience in ai coding are able to do more with ai and get more ambitious things done more quickly. A bummer is that we have a genai team (louie.ai) and a gpu/viz/graph analytics team (graphistry), and those who have spent the last 2-3 years doing genai daily have a higher uptake rate here than those who aren't. I wouldn't say team 1 is better than team 2 in general: these are tools, and different people have different engineering skill and ai coding skill, including different amounts of time doing both. What was a revelation for me personally was taking 1-2mo early in claude code's release was to go full cold turkey on manual coding, similar to getting immersed in a foreign language. That forced eliminating a lot of bad habits wrt effective ai coding both personally and in state of our repo tooling. Since then, it's been steady work to accelerate and smooth that loop, eg, moving from vibe coding/engineering to now more eval-driven ai coding loops: https://media.ccc.de/v/39c3-breaking-bots-cheating-at-blue-team-ctfs-with-ai-speed-runs https://media.ccc.de/v/39c3-breaking-bots-cheating-at-blue-t... . That takes a LOT of buildout.
• intended 9mo ago
  Thank you for providing data which can actually be used to collate! I strongly suspect that experience is a huge determinant of what utility is seen from LLMs. It seems that theres more people writing and finishing projets, but not many have reached the point where they have to maintain their code / deal with the tech debt.
• KronisLV 9mo ago
  > I end up rewriting about 70% of the thing. Doesn't match my experience, that figure is closer to about 20-40% to me, though a lot of those changes I want are possible by just further prompting OR turning to a different model, or adding some automated checks that promptly fail and the AI can do a few more loops of fixes. > Other people are just less picky than I am, or they have a less thorough review culture that lets subpar code slide more often. This is also likely, or you are just doing stuff that is worse represented in the training data, or working on novel things where the output isn't as good. But I'm leaning towards people just being picky about what they view as "good code" (or underspecifying how the AI is supposed to output it) at least roughly since Sonnet 4, since with some people I work with it's just endless and oftentimes meaningless discussions and bikeshedding when in code review. You can always be like: "This here pattern in these 20 files is Good Code™, use the same collection of approaches and code style when working on this refactoring/new feature."
  • 9dev 9mo ago
    > You can always be like: "This here pattern in these 20 files is Good Code™, use the same collection of approaches and code style when working on this refactoring/new feature." …and then add that to your CLAUDE.md, and never worry about having to say it again manually.
    • KronisLV 9mo ago
      Exactly! Unless you use something that doesn’t read CLAUDE.md, then you’d still just tell the model to read the file as a part of its work. What helped me a bunch was having prebuild scripts (can be Bash, can be Python, can be whatever) for each of the architectural or style conventions I want to enforce. Tools like ESLint are also nice but focused a bit more on the code than architecture/structure. Problems start when a colleague might just remove some of those due to personal preference without discussion but then you have other problems - in my experience, with proper controls in place AI will cause less issues and friction than people (ofc depending on culture fit).
• Closi 9mo ago
  Try learning to vibe code on something totally greenfield without looking at the code and see if it changes your mind. Ignore code quality, “does it work” and “am i happy with the app” are the only metrics. Code quality is an issue you need to ignore with vibe coding - if code quality is important to your project or you then it’s not an issue. But if you abandon this concept and build things small enough or modular enough then speed gains await! IMO codebases can be architected for LLMs to work better in them, but this is harder in brownfield apps.
  • ahtihn 9mo ago
    If you start greenfield and ignore the code quality, how do you know you can maintain it long term? Greenfield is fundamentally easier than maintaining existing software. Once software exists, users expect it to behave a certain way and they expect their data to remain usable in new versions. The existing software now imposes all sorts of contraints that may not be explicit in the spec. Some of these constraints end up making some changes very hard. Bad assumptions in data modeling can make migrations a nightmare. You can't just write entirely new software every time the requirements change.
    • Closi 9mo ago
      In practice, this is managed by: 1) Making the application small enough, and breaking it apart if needed (e.g. I've refactored my old 'big' app into 10 micro-apps). 2) Selecting an architecture that will work, looking after the data modelling and architecture yourself rather than delegating this to the LLM (it can implement it - but you need to design it). 3) Trusting that the LLM is capable enough to implement new requirements or fixes as required. If requirements change so substantially that it's not possible, you can write new software as requirements change - as per point 1, you will have made your application modular enough that this isn't a significant concern.
• rbbydotdev 9mo ago
  > I end up rewriting about 70% of the thing. I think this touches on the root of the issue. I am seeing a results over process winning. Code quality will reduce. Out of touch or apathetic project management who prioritize results, now are even more emboldened to have more tech debt riddled code
• AlexCoventry 9mo ago
  > I don't understand the stance that AI currently is able to automate away non-trivial coding tasks. I'm happy enough for it to automate away the trivial coding tasks. That's an immense force multiplier in its own right.
• Balinares 9mo ago
  That's been pretty much exactly my experience too. For what it's worth, multiple times in my career, I've worked at shops that once thought they could do it quick and cheap and it would be good enough, and then had to hire someone 'picky' like me to sort out the inevitable money-losing mess. From what I've seen even Opus 4.5 spit, the 'picky' are going to remain in demand for a little while longer still. Will that last? No clue. We'll see.
  • cpursley 9mo ago
    You can be picky with Opus, just yell at it to refactor a few times. To reduce refactor cycles, give it correct and enough context before you start along with expected code style, etc. These things aren't one shot magic machines.
• deleted 9mo ago
  [deleted]
• antirez 9mo ago
  After you review, instead of rewriting 70% of the code, have you tried to follow up with a message with a list of things to fix? Also: in my experience 1. and 2. are not needed for you to have bad results. The existing code base is a fundamental variable. The more complex / convoluted it is, the worse is the result. Also in my experience LLMs are constantly better at producing C code than anything else (Python included). I have the feeling that the simplicity of the code bases I produced over the years, and that now I modify with LLMs, and the fact they are mostly in C, is a big factor why LLMs appear to work so well for me. Another thing: Opus 4.5 for me is bad on the web, compared to Gemini 3 PRO / GPT 5.2, and very good if used with Claude Code, since it requires to reiterate to reach the solution, why the others sometimes are better first-shotter. If you generate code via the web interface, this could be another cause. There are tons of variables.
  • embedding-shape 9mo ago
    > After you review, instead of rewriting 70% of the code, have you tried to follow up with a message with a list of things to fix? I think this is the wrong approach, already by having "wrong code" in the context, makes every response after this worse. Instead, try restarting, but this time specify exactly how you expected that 70% of the code to actually have worked, from the get go. Often, LLMs seem to make choices because they have to, and if you think they made the wrong choice, you can often find that you didn't actually specify something well enough, hence the LLM had to do something, since apparently the single most important thing for them is that they finish something, no matter how right or wrong. After a while, you'll get better at knowing what you have to be precise, specific and "extra verbose" about, compared to other things. Something that also seems to depend on the model, like with how Gemini you can have 5 variations of "Don't add any comments" yet it does anyways, but say that once to GPT/Claude-family of models and it seems they get it at once.
    • notnullorvoid 9mo ago
      There are some problems where this becomes a game of whack-a-mole either way you approach it (restart or modify with existing context). I end up writing more prompts than the code I could've written myself. This isn't to say I don't think LLMs are an asset, they have helped me solve problems and grow in domains where I lack experience.
• crassus_ed 9mo ago
  Genuine question, doesn't this apply to coding style than actual results? Same applies to writing style. LLMs manage to write great stories but they don't suit my writing style. When generating code it doesn't always suit my coding style but the code it generates functions fine.
• nvarsj 9mo ago
  It is pretty simple imo. AI (just like humans!) does best on well written, self contained code bases. Which is a very small niche, but also over represented in open source and subsequently by tech celebrities who tend not to work on “ugly code”. I work on a giant legacy code base at big tech, which is one piece of many distributed systems. LLM is helpful for localised, well defined work, but nowhere close to what the TFA describes.
• egorfine 9mo ago
  Same experience. The better the model the more complicated are the bugs and brain damages it introduces. Perhaps one has to be skilled programmer in the first place to spot the problems, which is not easy when the program runs apparently. Things like mocked tests, you know. Who would care about that.
• javier2 9mo ago
  This is also my experience with enterprise Java. LLMs have done much better with slightly less convoluted code bases in Go. Its currently clearly better at Go and Typescript than Java in my view
• cm2187 9mo ago
  Not trying to back the AI hype, but most pre-AI auto generated code is garbage (like winform auto generated code or entity framework SQL in the .net world). But that’s fine, it’s not meant to be read by humans. If you want to change it you can regenerate it. It may be that AI just moves the line between what developers should care and look at vs the boring boiler plate code that has little value added.
  • camdenreslink 9mo ago
    But those code generators were deterministic (and indeed caused huge headaches if the generated code changed between versions). Seems like a totally different thing.
• eloisant 9mo ago
  You can definitely use AI for non-trivial tasks. It's not just about better prompting, but using better tools. Tools that will turn a bad prompt into a good prompt. For example there is the plan mode for Cursor. Or just ask the AI: "make a plan to do this task", then you review the plan before asking it to implement. Configure the AI to ask you clarification questions instead of assuming things. It's still evolving pretty quickly, so it's worth staying up to date with that.
  • xmodem 9mo ago
    I have not been as aggressive as GP in trying new AI tools. But the last few months I have been trying more and more and I'm just not seeing it. One project I tried out recently I took a test-driven approach. I built out the test suite while asking the AI to do the actual implementation. This was one of my more successful attempts, and may have saved me 20-30% time overall - but I still had to throw out 80% of what it built because the agent just refused to implement the architecture I was describing. It's at its most useful if I'm trying to bootstrap something new on a stack I barely know, OR if I decide I just don't care about the quality of the output. I have tried different CLI tools, IDE tools. Overall I've had the best success with Claude Code but I'm open to trying new things. Do you have any good resources you would recommend for getting LLM's to perform better, or staying up-to-date on the field in general?
    • eloisant 9mo ago
      If you haven't yet, check Claude Code's plan mode: https://claudelog.com/mechanics/plan-mode/ https://claudelog.com/mechanics/plan-mode/
• jstummbillig 9mo ago
  > Every single time [...] I end up rewriting about 70% of the thing If that number has not significantly changed since GPT 3.5, I think it's safe to assume that something very weird is happening on your end.
  • dns_snek 9mo ago
    I think I know what they mean, I share a similar experience. It has changed, 3.5 couldn't even attempt to solve non-trivial tasks so it was a 100% failure, now it's 70%.
• austin-cheney 9mo ago
  Like with anything else the people best positioned to enjoy output are the people least well positioned to criticize it. This is true of AI just as eating at restaurants or enjoying movie dramas.
• dspillett 9mo ago
  > 1) I'm not good at prompting, I assume this is part of the problem (though I've avoided using LLMs mostly so can't comment with any true confidence here) but to a large extent this is blaming you for a suboptimal interface when the interface is the problem. That some people seem to get much better results than others, and that the distinction does not map well to differences in ability elsewhere, suggests to me that the issue is people thinking slightly differently and the training data for the models somehow being biased to those who operate in certain ways. > 2) Other people are just less picky than I am That is almost certainly a much larger part of the problem. “Fuck it, it'll do, someone else can tidy it later if they are bothered enough” attitudes were rampant long before people started outsourcing work to LLMs.
• davidguetta 9mo ago
  I think you are not hardcore enough. I paste entire files or 2 3 files at once and ask to rewrite everything. Then you rewiew it and in general have to ask to remove some stuff. And then it's (good enough). You have to accept to not nitpick some parts (like random functions being generated) as long as your test suite pass, otherwise of course you will end up rewritin everything It also depends on your setting, some area (web vs AI vs robotics) can be more suited than other
• wanderlust123 9mo ago
  Do you have an example of something that was subpar and needed a 70% rewrite?
• PunchyHamster 9mo ago
  It's the second. Like, yes, prompting is a skill and you need to learn it for AI to do something useful but usefulness quickly falls down a cliff once you go past "greenfield implementation" or "basically example code" or "the thing done a lot so AI have a lot of reference to put from" it quickly gets into kinda sorta but not really working state. It can still be used effectively on smaller parts of the codebase (I used it a lot basically to generate some boilerplate to run the test even if I had to rewrite a bunch of actual tests) but as whole very, very overrated by the AI peddlers. And it probably stems from the fact that for the clueless ones it looks like amazing productivity boost because they go from "not even knowing framework" to "somewhat working app"
  • nicce 9mo ago
    People already say here that they don’t even look the code anymore. ”That is AIs job”. As long as there is a spec and tests pass, they are happy! I just can’t do that.
    • mixdup 9mo ago
      It's just the next rung on the enshittification ladder. So many steps in our "progress" to enlightenment as a society, as a technology community, is just abstracting away work with a "good enough" solution that is around an 80% solution That's fine for the first iteration or two, because you think "oh man this is going to make me so productive, I'll be able to use this new productivity to wring 40% of progress out of that 20% gap" But instead we just move on to the next thing, bring that 20% shittified gap along with us, and the next thing that gets built or paved over has a 20% gap, and eventually we're bankrupt from rolling over all that negative equity
      • nicce 9mo ago
        The counter argument for this is the comparison for traditional compilers. AI is "the new compiler", just for natural language. The optimization happens over time! But I am not so sure about that.
• prezk 9mo ago
  Maybe LLMs are like a next evolution of a rubber ducky: you can talk to it, and it's very helpful, just don't expect that IT will give you the final answer.
• f1shy 9mo ago
  I'm exactly on the same boat. To anybody who want to try, a concrete example, that I have tested in all available LLMs: Make a prompt to get a common lisp application which makes a "hello triangle" in open gl, without using SDL or any framework, only OpenGL and GLFW bindings. None of the replies even compiled. I kept asking at least 5 times, with error feedback, to see if AI can do it. It did't work. Never. The best I got was from gemini, a code where I had to change about 10 lines, absolutely no trivial changes that need to be familiar with opengl and lisp. After doing the changes I asked back, what does it think of the changes, it replied I was wrong, with those changes it will never work. If anybody can make a prompt that get me that, please let me known...
  • Philpax 9mo ago
    It sounds like you're using LLMs directly, instead of a coding agent. Agents are capable of testing their own code and using that to fix issues, which is what makes them so powerful. Using Claude Code, I was able to successfully produce the Hello Triangle you asked for (note that I have never used CL before): https://github.com/philpax/hello-triangle-cl https://github.com/philpax/hello-triangle-cl For reference, here is the transcript of the entire interaction I had with CC (produced with simonw's excellent claude-code-transcripts): https://gisthost.github.io/?7924519b32addbf794c17f4dc7106bc2/index.html https://gisthost.github.io/?7924519b32addbf794c17f4dc7106bc2... Edit: To better contextualise what it's doing, the detailed transcript page may be useful: https://gisthost.github.io/?7924519b32addbf794c17f4dc7106bc2/page-001.html https://gisthost.github.io/?7924519b32addbf794c17f4dc7106bc2...
    • f1shy 9mo ago
      Nice. The code I got running from gemini was much much cleaner, it did not work, but after the manual changes it did work. I will hive it a try with the next task: put text and generate primitives like rectangle, circle, polygon, etc…
  • james_a_craig 9mo ago
    "Please write me a program in Common LISP (SBCL is installed) which will render a simple "hello world" triangle in OpenGL. You should use only OpenGL and GLFW (using sbcl's FFI) for this, not any other existing 3D graphics framework." This worked in codex-cli, albeit it took three rounds of passing back the errors. https://gist.github.com/jamesacraig/9ae0e5ed8ebae3e7fe157f67ceeec2e4 https://gist.github.com/jamesacraig/9ae0e5ed8ebae3e7fe157f67... has the resulting code.
• torginus 9mo ago
  Personally I've yet to see any high profile programming person (who's not directly invested into AI) endorse only coding by prompting. Experienced coders that I follow, who do use AI tend to focus on tight and fast feedback loops, and precise edits (or maybe exploratory coding) rather than agentic fire-and-forget workflows. Also, an interesting side note, I expected programmers I think of as highly skilled, who I know personally to reject AI from personal pride - that has not been the case. However 2 criticisms I've heard consistently from this crowd (besides the thing I mentioned before) was - AI makes hosting and participating in coding competitions impossible, and denies them of brain-teasers and an ability to hone their skills. - A lot of them are concerned about the ethics of training on large codebases - and consider AI plagiarism as much of an issue as artists do.
• mark_l_watson 9mo ago
  I think you are correct, with one large caveat: With very good tooling (e.g., Google Antigravity, Claude Coding, Open AI’s codex, and several open platforms) and not caring about your monthly API and subscription costs, then very long running trial and error and also with tools for testing code changes, then some degree of real autonomy is possible. But, do we want to work like this? I don’t. I feel very good about using strong AI for research and learning new things (self improvement) and I also feel good about using strong AI as a ‘minor partner’ in coding.
• zelphirkalt 9mo ago
  That's the curse of the expert. You see many of the shortcomings, that someone less experienced might not even think about, when they go to social media and blurt out that AI is now able to fully replace them.
• ninininino 9mo ago
  How much buggy / incorrect Java written by first year computer science University students is there on Stack Overflow (in SO post bodies)? Decades of it. Ask the same question of Golang, or Rust, or Typescript. I have a theory that the large dichotomy in how people experience AI coding has to do with the quality of the training corpus for each language online.
• teunispeters 9mo ago
  Wow only 70%. I so far have had to drop and rewrite from scratch every time. Mind, I work in C/embedded spaces, and current LLMs are just horrible at any code in that space. My vote is with (2).
• parliament32 9mo ago
  Not even coding tasks. Just getting an LLM to help me put together a PromQL query to do something somewhat non-standard takes dozens of tries and copy/pasting back error messages.. and these aren't complex errors, trivial things like missing closing brackets and the like. I know the usual clap back is "you're just missing this magical workflow" or "you need to prompt better" but.. do I really need to prompt "make sure your syntax is correct"? Shouldn't that be, ya know, a given for a prompt that starts with "Help me put together a PromQL query that..."?
  • simonw 9mo ago
    Yes, you're missing a magic workflow. If you find yourself having to copy and paste errors back and forward you need to upgrade to a coding agent harness like Claude Code so the LLM can try things out and then fix the errors on its own. If you're not willing to do that you can also fix this by preparing a text file with a few examples of correctly formatted queries and pasting that in at the start of your session, or putting it in a skill markdown file.
    • parliament32 9mo ago
      So, let me get this straight, LLMs need a "coding agent harness" to figure out that they need to close brackets? Wild.
      • simonw 9mo ago
        They need one if you want them to be able to automatically recover from mistakes they make, yes.
• hansvm 9mo ago
  You alluded to it, but also: 3) Not everyone codes the same things 4) It's easy to get too excited about the tech and ignore its failure modes when describing your experiences later I use AI a lot. With your own control plane (as opposed to a generic Claude Code or whatever) you can fully automate a lot more things. It's still fundamentally incapable of doing tons of tasks though at any acceptable quality level, and I strongly suspect all of (2,3,4) are guiding the disconnect you're seeing. Take the two things I've been working on this morning as an example. One was a one-off query. I told it the databases it should consider, a few relevant files, roughly how that part of the business works, and asked it to come back when it finished. When it was done I had it patch up the output format. It two-shot (with a lot of helpful context) something that would have taken me an hour or more. Another is more R&D-heavy. It pointed me to a new subroutine I needed (it couldn't implement it correctly though) and is otherwise largely useless. It's actively harmful to have it try to do any of the work. It's possible that (1) matters more than you suspect too. AI has certain coding patterns it likes to use a lot which won't work in my codebase. Moreover, it can't one-shot the things I want. It can, however, follow a generic step-by-step guide for generating those better ideas, translating worse ideas into things that will be close enough to what I need, identifying where it messed up, and refactoring into something suitable, especially if you take care to keep context usage low and whatnot. A lot of people seem to be able to get away with CLAUDE.md or whatever, but I like having more granular control of what the thing is going to be doing.
• locknitpicker 9mo ago
  > Every single time, I get something that works, yes, but then when I start self-reviewing the code, preparing to submit it to coworkers, I end up rewriting about 70% of the thing. You might want to review how you approach these tools. Complaining that you need to rewrite 70% of the code screams of poor prompting, with too vague inputs, no constraints, and no feedback at all. Using agents to help you write code is far from a one-shot task, but if throwing out 70% of what you create screams out that you are prompting the agent to create crap. > 1) I'm not good at prompting, even though I am one of the earliest AI in coding adopters I know, and have been consistent for years. So I find this hard to accept. I think you need to take a humble pill, review how you are putting together these prompts, figure out what you are doing wrong in prompts and processes, and work up from where you are at this point. If 70% of your output is crap, the problem is in your input. I recommend you spend 20 minutes with your agent of choice prompting it to help you improve your prompts. Check instruction files, spec-driven approaches, context files, etc. Even a plain old README.md helps a lot. Prompt your agent to generate it for you. From there, instead of one-shot prompts try to break down a task into multiple sub steps with small deliverables. Always iterate on your instruction files. It you spend a few minutes on this, you will quickly halve your churn rate.
• alexsmirnov 9mo ago
  This is exact the impression that I got. Every question or task given to LLM returns pretty reasonable, but flawed result. For the coding, those are hard to spot but dangerous mistakes. They all look good and perfectly reasonable, but just wrong. Anthropic compared Claude Code to a "slot machine", and I fell that AI coding now is something close to gambling addiction. As small wins keep gambler to make more bets, so correct results from AI keep developers to use it: "I see it made correct solution, let's try again!" At a startup CTO, I review most of the pull requests from team members, and team uses AI tools actively. The overall picture strongly confirms your second conclusion.
  • simonw 9mo ago
    If someone gives you access to a slot machine which is weighted such that it pays out way more than you put into it, my advice is to start cranking that lever. If it does indeed start costing more than it's paying out, step away.
• zmj 9mo ago
  My experience with agents in larger / older codebases is that feedback loops are critical. They'll get it somewhere in the neighborhood of right on the first attempt; it's up to your prompt and tooling to guide them to improve it on correctness and quality. Basic checks: can the agent run the app, interact with it, and observe its state? If not, you probably won't get working code. Quality checks: by default, you'll get the same code quality as the code the agent reads while it's working; if your linters and prompts don't guide it towards your desired style, you won't get it. To put that another way: one-shots attempts aren't where the win is in big codebases. Repeat iteration is, as long as your tooling steers it in the right direction.
• perrygeo 9mo ago
  LLMs tend to rise to the level of the complexity of the codebase. They are probabilistic pattern matching machines, after all. It's rare to have a 15 year old repo without significant complexity; is it possible that the reason LLMs have trouble with complex codebases is that the codebases are complex? IMO it has nothing to do with LLMs. They just mirror the patterns they see - don't get upset when you don't like your own reflection! Software complexity is still bad. LLMs just shove it back in our face. Implications: AI is always going to feel more effective on brand new codebases without any legacy weight. And less effective on "real" apps where the details matter. The bias is strongly evident - you rarely hear anyone talking about how they vibe coded a coherent changeset to an existing repo.
• h14h 9mo ago
  In my experience, using AI coding agents need highly specific success criteria, and an easy way to verify its output against that criteria. My biggest successes have come when I take a TDD approach. First I identify a subset of my work into a module with an API that can be easily tested, then I collaborate with the agent on writing correct test-cases, and finally I tell it to implement the module such that the test cases pass without any lint or typing errors. It forces me to spend much more time thinking about use cases, project architecture, and test coverage than about nitty-gritty implementation details. I can imagine that in a system that evolved over time without a clear testing strategy, AI would struggle mightily to be even barely useful. Not saying this applies to your system, but I've definitely worked on systems in the past that fit the "big ball of mud" description pretty neatly, and I have zero clue how I'd have been able to make effective use of these AI tools.