41 ms·

Why LLMs can't really build software

• deleted 1y ago
  [deleted]
• 9cb14c1ec0 1y ago
  > what they cannot do is maintain clear mental models The more I use claude code, the more frustrated I get with this aspect. I'm not sure that a generic text-based LLM can properly solve this.
  • cmrdporcupine 1y ago
    Honestly it forces you -- rightfully -- to step back and be the one doing the planning. You can let it do the grunt coding, and a lot of the low level analysis and testing, but you absolutely need to be the one in charge on the design. It frankly gives me more time to think about the bigger picture within the amount of time I have to work on a task, and I like that side of things. There's definitely room for a massive amount of improvement in how the tool presents changes and suggestions to the user. It needs to be far more interactive.
    • mock-possum 1y ago
      That’s my experience as well - I’m the one with the mental model, my responsibility is using text to communicate that model to the LLM using language it will recognize from its training data to generate the code to follow suit. My experience with prompting LLMs for codegen is really not much different from my experience with querying search engines - you have to understand how to ‘speak the language’ of the corpus being searched, in order to find the results you’re looking for.
    • micromacrofoot 1y ago
      Yes this is exactly it, you need to talk to Claude about code on a design/architecture level... just telling it what you want the code to output will get you stuck in failure loops. I keep saying it and no one really listens: AI really is advanced autocomplete. It's not reasoning or thinking. You will use the tool better if you understand what it can't do. It can write individual functions pretty well, stringing a bunch of them together? not so much. It's a good tool when you use it within its limitations.
  • dlivingston 1y ago
    Reminds me of how Google's Genie 3 can only run for a ~minute before losing its internal state [0]. My gut feeling is that this problem won't be solved until some new architecture is invented, on the scale of the transformer, which allows for short-term context, long-term context, and self-modulation of model weights (to mimic "learning"). (Disclaimer: hobbyist with no formal training in machine learning.) [0]: https://news.ycombinator.com/item?id=44798166 https://news.ycombinator.com/item?id=44798166
    • skydhash 1y ago
      It’s the nature of formal system. Someones need to actually do the work of defining those rules or have a smaller set of rules that can generate the larger set. But anytime you invent a rule. That means a few things that are possible can’t be represented in the system. You’re mostly hoping that those things aren’t meaningful. LLMs techniques allows us to extract rules from text and other data. But those data are not representative of a coherent system. The result itself is incoherent and lacks anything that wasn’t part of the data. And that’s normal. It’s the same as having a mathematical function. Every point that it maps to is meaningful, everything else may as well not exists.
  • elephanlemon 1y ago
    I’ve been thinking about this recently… maybe a more workable solution at the moment is to run a hierarchy of agents, with the top level one maintaining the general mental model (and not filling its context with anything much more than “next agent down said this task was complete”). Definitely seems like anytime you try to have one Code agent run everything it just goes off the rails sooner or later, ignoring important details from your original instructions, failing to make sure it’s adhering to CLAUDE.md, etc. I think you can do this now with Code’s agent feature? Anyone have strategies to share?
    • skydhash 1y ago
      Telephone game don’t work that well. That’s how an emperor can be isolated in his palace and every edict becomes harmful. It’s why architect/developer didn’t work. You need to be aware of all the context you need to make sure you’ve done a good job
  • edaemon 1y ago
    Same here. I have used this tool which helps a bit: https://github.com/rizethereum/claude-code-requirements-builder https://github.com/rizethereum/claude-code-requirements-buil... That and other tricks have only made me slightly less frustrated, though.
  • SoftTalker 1y ago
    Is this really that diffferent from the "average" programmer, especially a more junior one? > LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. I see this constantly with mediocre developers. Flailing, trying different things, copy-pasting from StackOverflow without understanding, ultimately deciding the compiler must have a bug, or cosmic rays are flipping bits.
    • layer8 1y ago
      The article explicitly calls out that that’s what they are looking for in a competent software engineer. That incompetent developers exist, and that junior developers tend to not be very competent yet, doesn’t change anything about that. The problem with LLMs is that they’re already the final product of training/learning, not the starting point. The (in)ability of an LLM to form stable mental models is fixed in its architecture, and isn’t anything you can teach them.
      • SoftTalker 1y ago
        I just (re) read the article and the word "competent" doesn't appear in it. It doesn't discuss human developer competency at all, except in comparison to LLMs.
        • layer8 1y ago
          Yes, I replaced “effective” by “competent” in my response, because I found that word slightly preferable in the context discussed.
    • Xss3 1y ago
      I feel like something is wrong where you are, maybe your juniors do not feel incentivized or encouraged to learn, code reviews might not be strict enough, quality may not be valued enough, and immense pressure to move tickets might be put on people, or all of the above in various doses. I feel this way because at my company our interns on a gap year from their comp sci degree don't blame the compiler, cosmic bits, or blindly copy from stack overflow. They're incentivized and encouraged to learn and absolutely choose to do so. The same goes for seniors. If you say 'I've been learning about X for ticket Y' in the standup people basically applaud it, managers like us training ourselves to be better. Sure managers may want to see a brief summary or a write-up applicable to our department if you aren't putting code down for a few days, but that's the only friction.
• empath75 1y ago
  It's good at micro, but not macro. I think that will eventually change with smarter engineering around it, larger context windows, etc. Never underestimate how much code that engineers will write to avoid writing code.
  • pmdr 1y ago
    > It's good at micro, but not macro. That's what I've found as well. Start describing or writing a function, include the whole file for context and it'll do its job. Give it a whole codebase and it will just wander in the woods burning tokens for ten minutes trying to solve dependencies.
• usrbinbash 1y ago
  > We don't just keep adding more words to our context window, because it would drive us mad. That, and we also don't only focus on the textual description of a problem when we encounter a problem. We don't see the debugger output and go "how do I make this bad output go away?!?". Oh, I am getting an authentication error. Well, meaybe I should just delete the token check for that code path...problem solved?! No. Problem very much not-solved. In fact, problem very much very bigger big problem now, and [Grug][1] find himself reaching for club again. Software engineers are able to step back, think about the whole thing, and determine the root cause of a problem. I am getting an auth error...ok, what happens when the token is verified...oh, look, the problem is not the authentication at all...in fact there is no error! The test was simply bad and tried to call a higher privilege function as a lower privilege user. So, test needs to be fixed. And also, even though it isn't per-se an error, the response for that function should maybe differentiate between "401 because you didn't authenticate" and "401 because your privileges are too low". [1]: https://grugbrain.dev https://grugbrain.dev
  • trod1234 1y ago
    Isn't the 401 for LLMs the same single undecidable token? Doesn't this basically go to the undecidable nature of math in CS? Put another way, you have an excel roster corresponding to people with accounts where some need to have their account shutdown but you only have their first and last names as identifiers, and the pool is sufficiently large that there are more than one person per a given set of names. You can't shut down all accounts with a given name, and there is no unique identifier. How do you solve this? You have to ask and be given that unique identifier that differentiates between the undecidable. Without that, even the person can't do the task. The person can make guesses, but those guesses are just hallucinations with a significant n probability towards a bad repeat outcome. At a core level I don't think these type of issues are going to be solved. Quite a lot of people would be unable to solve this and struggle with this example (when not given the answer, or hinted at the solution in the framing of the task; ie when they just have a list of names and are told to do an impossible task).
  • livid-neuro 1y ago
    The first cars broke down all the time. They had a limited range. There wasn't a vast supply of parts for them. There wasn't a vast industry of experts who could work on them. There wasn't a vast network of fuel stations to provide energy for them. The horse was a proven method. What an LLM cannot do today is almost irrelevant in the tide of change upon the industry. The fact is, with improvements, it doesn't mean an LLM cannot do it tomorrow.
• jmclnx 1y ago
  I am not a fan of today's concept of "AI", but to be fair, building today's software is not for the faint of heart, very few people gets it right on try 1. Years ago I gave up compiling these large applications all together. I compiled Firefox via FreeBSD's (v8.x) ports system, that alone was a nightmare. I cannot imagine what it would be like to compile GNOME3 or KDE or Libreoffice. Emacs is the largest thing I compile now.
  • anotherhue 1y ago
    I suggest trying Nix, by being reproducible those nasty compilation demons get solved once and for all. (And usually by someone else)
    • trod1234 1y ago
      The problem with Nix is that its often claimed to be reproducible, but the proof isn't really there because of the existence of collisions. The definition of reproducible is taken in such an isolated context as to be almost absurd. While a collision hasn't yet been found for a SHA256 package on Nix, by the pigeonhole principle they exist, and the computer will not be able to decide between the two packages in such a collision leading to system level failure, with errors that have no link to cause (due to the properties involved, and longstanding CS problems in computation). These things generally speaking contain properties of mathematical chaos which is a state that is inherently unknowable/unpredictable that no admin would ever approach or touch because its unmaintainable. The normally tightly coupled error handling code is no longer tightly coupled because it requires matching a determinable state (CS computation problems, halting/decidability). Non-deterministic failure domains are the most costly problems to solve because troubleshooting which leverages properties of determinism, won't work. This leaves you only a strategy of guess and check; which requires intimate knowledge of the entire system stack without abstractions present.
      • anotherhue 1y ago
        Respectfully, you sound like AI. I expect you don't trust git either, especially as its hash is weaker. A cursory look at a nix system would also show you that the package name, version and derivation sha are all concatenated together.
• emilecantin 1y ago
  Yeah, I think it's pretty clear to a lot of people that LLMs aren't at the "build me Facebook, but for dogs" stage yet. I've had relatively good success with more targeted tasks, like "Add a modal that does this, take this existing modal as an example for code style". I also break my problem down into smaller chunks, and give them one by one to the LLM. It seems to work much better that way.
  • hahn-kev 1y ago
    I do wonder how something like v0 would handle that request though.
  • bagacrap 1y ago
    I can already copy paste existing code and tweak it to do what I want (if you even consider that "software engineering"). The difference being that my system clipboard is deterministic, rather than infinitely creative at inventing new ways to screw up.
• deleted 1y ago
  [deleted]
• trod1234 1y ago
  I think most people trying to touch on this topic don't consider this byline with other similar bylines like, "Why LLMs can't recognize themselves looping", or "Why LLMs can't express intent", or "Why LLMs can't recognize truth/falsity, or confidence levels of what they know vs don't know", these other bylines basically with a little thought equate to Computer Science halting problems, or the undecidability nature of mathematics. Taken to a next step, recognizing this makes the investment in such a moonshot pipedream (overcoming these inherent problems in a deterministic way), recklessly negligent.
• deleted 1y ago
  [deleted]
• saghm 1y ago
  > Context omission: Models are bad at finding omitted context. > Recency bias: They suffer a strong recency bias in the context window. > Hallucination: They commonly hallucinate details that should not be there. To be fair, those are all issues that most human engineers I've worked with (including myself!) have struggled with to various degrees, even if we don't refer to them the same way. I don't know about the rest of you, but I've certainly had times where I found out that an important nuance of a design was overlooked until well into the process of developing something, forgotten a crucial detail that I learned months ago that would have helped me debug something much faster than if I had remembered it from the start, or accidentally make an assumption about how something worked (or misremembered it) and ended up with buggy code as a result. I've mostly gotten pretty positive feedback about my work over the course of my career, so if I "can't build software", I have to worry about the companies that have been employing me and my coworkers who have praised my work output over the years. Then again, I think "humans can't build software reliably" is probably a mostly correct statement, so maybe the lesson here is that software is hard in general.
  • skydhash 1y ago
    That’s a communication issue. You should learn how to ask the right questions and document the answers given. What I’ve seen is developers assuming stuff when they should just reach out to team members. Or trying stuff instead of reading documentation. Or trying to remember info instead of noting it down somewhere.
    • saghm 1y ago
      Well, yeah, obviously if you're perfectly diligent and never screw up, it's possible to be correct 100% of the time. In my experience, even extremely smart diligent people who are good at asking the right questions and reading documentation still mess up sometimes, which is the point I'm trying to make. If you genuinely don't ever encounter this issue, I guess everyone I've ever worked with and I just aren't as perfect as you and the people you've worked with, but I'd argue that you're not having the average experience of working with regular people if that's the case. Most of us are mere mortals who are sometimes fallible, and while the exact underlying mechanism of how we make mistakes might not be literally identical to the issues described in the article, my point is that the difference might just be a matter of degree rather than something fundamentally different in what types of errors occur.
• myfavoritedog 1y ago
  [dead]
• Transfinity 1y ago
  > LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. I feel personally described by this statement. At least on a bad day, or if I'm phoning it in. Not sure if that says anything about AI - maybe just that the whole "mental models" part is quite hard.
  • apples_oranges 1y ago
    It means something is not understood. Could be the product, the code in question, or computers in general. 90% of coders seem to be lacking foundational knowledge imho. Not trying to hate on anyone, but when you have the basics down, you can usually see quickly where the problem is, or at least must be.
    • aniviacat 1y ago
      Unfortunately, "usually" is a key word here.
  • bagacrap 1y ago
    So LLMs are always phoning it in, on a bad day, etc. Great. I recently tried to get AI to refactor some tests, which it proceeded to break. Then it iterated a bit till it had gotten the pass rate back up to 75%. At this point it declared victory. So yes, it does really seem like a human who really doesn't want to be there.
• Nickersf 1y ago
  I think they're another tool in the toolbox not a new workshop. You have to build a good strategy around LLM usage when developing software. I think people are naturally noticing that and adapting.
• generalizations 1y ago
  These LLM discussions really need everyone to mention what LLM they're actually using. > AI is awesome for coding! [Opus 4] > No AI sucks for coding and it messed everything up! [4o] Would really clear the air. People seem to be evaluating the dumbest models (apparently because they don't know any better?) and then deciding the whole AI thing just doesn't work.
  • omnicognate 1y ago
    What the article says is as true of Opus 4 as any other LLM.
  • troupo 1y ago
    > These LLM discussions really need everyone to mention what LLM they're actually using. They need to mention significantly more than that: https://dmitriid.com/everything-around-llms-is-still-magical-and-wishful-thinking https://dmitriid.com/everything-around-llms-is-still-magical... --- start quote --- Do we know which projects people work on? No Do we know which codebases (greenfield, mature, proprietary etc.) people work on? No Do we know the level of expertise the people have? No. Is the expertise in the same domain, codebase, language that they apply LLMs to? We don't know. How much additional work did they have reviewing, fixing, deploying, finishing etc.? We don't know. --- end quote --- And that's just the tip of the iceberg. And that is an iceberg before we hit another one: that we're trying to blindly reverse engineer a non-deterministic blackbox inside a provider's blackbox
  • taormina 1y ago
    I've used a wide variety of the "best" models, and I've mostly settled on Opus 4 and Sonnet 4 with Claude Code, but they don't ever actually get better. Grok 3-4 and GPT4 were worse, but like, at a certain point you don't get brownie points for not tripping over how low the bar is set.
    • generalizations 1y ago
      People have actually been basing their assertions on 4o. The bar is really low and people are still completely missing it.
• JimDabell 1y ago
  LLMs can’t build software because we are expecting them to hear a few sentences, then immediately start coding until there’s a prototype. When they get something wrong, they have a huge amount of spaghetti to wade through. There’s little to no opportunity to iterate at a higher level before writing code. If we put human engineering teams in the same situation, we’d expect them to do a terrible job, so why do we expect LLMs to do any better? We can dramatically improve the output of LLM software development by using all those processes and tools that help engineering teams avoid these problems: https://jim.dabell.name/articles/2025/08/08/autonomous-software/ https://jim.dabell.name/articles/2025/08/08/autonomous-softw...
  • diwank 1y ago
    yup. I started a fully autonomous, 100% vibe coded side project called steadytext, mostly expecting it to hit a wall, with LLMs eventually struggling to maintain or fix any non-trivial bug in it. turns out I was wrong, not only has claude opus been able to write up a pretty complex 7k LoC project with a python library, a CLI, _and_ a postgres extension. It actively maintains it and is able to fix filed issues and feature requests entirely on its own. It is completely vibe coded, I have never even looked at 90% of the code in that repo. it has full test coverage, passes CI, and we use it in production! granted- it needs careful planning for CLAUDE.md and all issues and feature requests need a lot of in-depth specifics but it all works. so I am not 100% convinced by this piece. I'd say it's def not easy to get coding agents to be able to manage and write software effectively and specially hard to do so in existing projects but my experience has been across that entire spectrum. I have been sorely disappointed in coding agents and even abandoned a bunch or projects and dozens of pull requests but I have also seen them work. you can check out that project here: https://github.com/julep-ai/steadytext/ https://github.com/julep-ai/steadytext/
    • sjdbdjskbzba 1y ago
      > It is completely vibe coded, I have never even looked at 90% of the code in that repo. it has full test coverage, passes CI, and we use it in production! This horrifies me. I checked your website and all your recommendations are from people who appear to have an Indian background, but you’re based in the US? And you claim they’re the most innovative companies yet I doubt anyone has heard of them? Looking over the repo and it seems like a mess (commits are meaningless and code is all over the place). I’m sorry this feels incredibly scammy.
• lordnacho 1y ago
  I think I agree with the idea that LLMs are good at the junior level stuff. What's happened for me recently is I've started to revisit the idea that typing speed doesn't matter. This is an age-old thing, most people don't think it really matters how fast you can type. I suppose the steelman is, most people think it doesn't really matters how fast you can get the edits to your code that you want. With modern tools, you're not typing out all the code anyway, and there's all sorts of non-AI ways to get your code looking the way you want. And that doesn't matter, the real work of the engineer is the architecture of how the whole program functions. Typing things faster doesn't make you get to the goal faster, since finding the overall design is the limiting thing. But I've been using Claude for a while now, and I'm starting to see the real benefit: you no longer need to concentrate to rework the code. It used to be burdensome to do certain things. For instance, I decided to add an enum value, and now I have to address all the places where it matches on that enum. This wasn't intellectually hard in the old world, you just got the compiler to tell you where the problems were, and you added a little section for your new value to do whatever it needed, in all the places it appeared. But you had to do this carefully, otherwise you would just cause more compile/error cycles. Little things like forgetting a semicolon will eat a cycle, and old tools would just tell you the error was there, not fix it for you. LLMs fix it for you. Now you can just tell Claude to change all the code in a loop until it compiles. You can have multiple agents working on your code, fixing little things in many places, while you sit on HN and muse about it. Or perhaps spend the time considering what direction the code needs to go. The big thing however is that when you're no longer held up by little compile errors, you can do more things. I had a whole laundry list of things I wanted to change about my codebase, and Claude did them all. Nothing on the business level of "what does this system do" but plenty of little tasks that previously would take a junior guy all day to do. With the ability to change large amounts of code quickly, I'm able to develop the architecture a lot faster. It's also a motivation thing: I feel bogged down when I'm just fixing compile errors, so I prioritize what to spend my time on if I am doing traditional programming. Now I can just do the whole laundry list, because I'm not the guy doing it.
  • spopejoy 1y ago
    A lot of this sounds like Claude is making up for bad tooling? Machine refactoring has been a thing for decades now but only for mainstream strongly-typed languages in major IDEs. You can do massive refactorings in seconds that always compile perfectly. LLMs are picking up the slack for js/ts, which is great, but pretty sad that we need this much compute to do something that took seconds 20 years ago in other languages.
• revskill 1y ago
  They can read and mind the error then figure out the best way to resolve. It is the best part about llm. No human can do it better than an llm. But they are not your mind reader. It is where things fall apart.
• chollida1 1y ago
  Most of this might be true for LLM's but years of investing experience has created a mental model of looking for the tech or company that sucks and yet keeps growing. People complained endlessly about the internet in the early to mid 90s, its slow, static, most sites had under construction signs on them, your phone modem would just randomly disconnect. The internet did suck in alot of ways and yet people kept using it. Twitter sucked in the mid 2000s, we saw the fail whale weekly and yet people continued to use it for breaking news. Electric cars sucked, no charging, low distance, expensive and yet no matter how much people complain about them they kept getting better. Phones sucked, pre 3G was slow, there wasn't much you could use them for before app stores and the cameras were potato quality and yet people kept using them while they improved. Always look for the technology that sucks and yet people keep using it because it provides value. LLM's aren't great at alot of tasks and yet no matter how much people complain about them, they keep getting used and keep improving through constant iteration. LLM"s amy not be able to build software today, but they are 10x better than where they were in 2022 when we first started using chatgpt. Its pretty reasonable to assume in 5 years they will be able to do these types of development tasks.
  • ausbah 1y ago
    those are really good points, but LLMs have really started to plateau off on their capabilities haven’t they? the improvements from gpt2 class models to 3 was much bigger then 3 to 4, which was only somewhat bigger than 4 to 5 most of the vibe shift I think I’ve seen in the past few months to using LLMs in the context of coding has been improvements in dataset curation and ux, not fundamentally better tech
    • worldsayshi 1y ago
      > LLMs have really started to plateau That doesn't seem unexpected. Any technological leap seem to happen in sigmoid-like steps. When a fruitful approach is discovered we run to it until diminishing returns sets in. Often enough a new approach opens doors to other approaches that builds on it. It takes time to discover the next step in the chain but when we do we get a new sigmoid-like leap. Etc...
• antihipocrat 1y ago
  ..."(at least for now) you are in the drivers seat, and the LLM is just another tool to reach for." Improvements in model performance seem to be approaching the peak rather than demonstrating exponential gains. Is the quote above where we land in the end?
• sneak 1y ago
  Am I the only one continuously astounded at how well Opus 4 actually does build mental models when prompted correctly? I find Sonnet frequently loses the plot, but Opus can usually handle it (with sufficient clarity in prompting).
• pjmlp 1y ago
  Only because most AI startups are doing it wrong. I don't want a chat window. I want AI workflows as part of my IDE, like Visual Studio, InteliJ, Android Studio are finally going after. I want voice controlled actions on my native language. Knowledge across everything on the project for doing code refactorings, static analysis with AI feedback loop, generating UI based out of handwritten sketches, programming on the go using handwriting, source control commit messages out of code changes,...
  • SunlitCat 1y ago
    Recently, i found lots of fun with codex-cli of openai, which even works as a mere ChatGPT Plus subscriber. :)
• codr7 1y ago
  Well, welcome to the club of awareness :)
  • layer8 1y ago
    Awareness is all we need. ;)
• Onewildgamer 1y ago
  I wonder if some of this can be solved by removing some wrongly setup context in LLM. Or get a short summary, restructure it and againt feed to a fresh LLM context.
  • layer8 1y ago
    I suspect that context can’t fully replace a mental model, because context is in-band, in the same band, as all input the LLM receives. It’s all just a linear token sequence that is taken in uniformly. There’s too little structure, and everything is equally subject to being discarded or distorted within the model. Even if parts of that token sequence remains unchanged (a “stable” context) when iterating over input, the input it is surrounded with can have arbitrary downstream effects within the model, making it more unreliable and unstable than mental models are.
    • Onewildgamer 1y ago
      Okay I see now. I'm just shooting in the dark here, if there's an ability to generate the next best token based on the trained set of words. Can it be taken a level up, in a meta level to generate a generation? like genetic programming does. Or is that what the chain of thought reasoning models do? Maybe I need to do more homework on LLMs in general.
• Xplan 1y ago
  [dead]
• jeffWrld 1y ago
  [dead]
• 1zael 1y ago
  > "when test fail, they are left guessing as to whether to fix the code or the tests" I've one thing that helps is using the "Red-Green-Refactor" language. We're in RED phase - test should fail. We're in GREEN phase - make this test pass with minimal code. We're in REFACTOR phase - improve the code without breaking tests. This helps the LLM understand the TDD mental model rather than just seeing "broken code" that needs fixing.
  • cortesoft 1y ago
    Sure, but isn't the issue when you are trying to move from the RED phase to the GREEN phase... are you still getting red because the test was bad or because the code isn't working yet?
• mccoyb 1y ago
  This is a low information density blog post. I’ve really liked Zed’s blog posts in the past (especially about the editor internals!) so I hope this doesn’t come the wrong way, but this seems to be a loose restatement of what many people are empirically finding out by using LLM agents. Perhaps good for someone just getting their feet wet with these computational objects, but not resolving or explaining things in a clear way, or highlighting trends in research and engineering that might point towards ways forward. You also have a technical writing no no where you cite a rather precise and specific study with a paraphrase to support your claims … analogous to saying “Godel’s incompleteness theorem means _something something_ about the nature of consciousness”. A phrase like: “Unfortunately, for now, they cannot (beyond a certain complexity) actually understand what is going on” referencing a precise study … is ambiguous and shoddy technical writing — what exactly does the author mean here? It’s vague. I think it is even worse here because _the original study_ provides task-specific notions of complexity (a critique of the original study! Won’t different representations lead to different complexity scaling behavior? Of course! That’s what software engineering is all about: I need to think at different levels to control my exposure to complexity)
• ontigola 1y ago
  Great, concise article. Nothing important to add, except that AI snake-oil salesmen will continue spreading their exaggerations far and wide, at least we who are truly in this business agree on the facts.
• nowittyusername 1y ago
  Saying LLMS are not good at x or y, is akin to saying a brain is useless without a body. Which is obvious. The success of agentic coding solutions depends on not just the model but also the system that the developers built around the model. And the companies that will succeed in this area are going to be the companies that focus on building sophisticated and capable systems that utilize said models. We are still in very early days where most organizations are only coming to terms with this realization... Only a few of them fully utilize this concept to the fullest, Claude code being the best example. The Claude models are specifically trained for tool calling and other capabilities and the Claude code cli compliments and takes advantage of those capabilities to the fullest, things like context management among other capabilities are extremely important ...
• alliancedamages 1y ago
  > ...but the distinguishing factor of effective engineers is their ability to build and maintain clear mental models. I wonder is this not just a proxy for intelligence?
• cindyllm 1y ago
  [dead]
• nextworddev 1y ago
  60% of the complaints in this post can be solved by providing better requirements and context upfront
  • windward 1y ago
    You can't just pile on more context, it has to be a brief distillation of the right context. Otherwise as it gets longer and longer, each point becomes less likely to be considered. Or,r worse, it gets considered in the negative. It feels at times like training a dog. You can't say "don't sit"! It's understandably frustrating that the promised future ended up being humans having to work how machines want.
    • nextworddev 1y ago
      Knowing what context to provide is the alpha.
• andrewmutz 1y ago
  The author does not understand what LLMs and coding tools are capable of today. > LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. This is exactly the opposite of what I am looking for. Software engineers test their work as they go. When tests fail, they can check in with their mental model to decide whether to fix the code or the tests, or just to gather more data before making a decision. When they get frustrated, they can reach for help by talking things through. And although sometimes they do delete it all and start over, they do so with a clearer understanding of the problem. My experiences are based on using Cline with Anthropic Sonnet 3.7 doing TDD on Rails, and have a very different experience. I instruct the model to write tests before any code and it does. It works in small enough chunks that I can review each one. When tests fail, it tends to reason very well about why and fixes the appropriate place. It is very common for the LLM to consult more code as it goes to learn more. It's certainly not perfect but it works about as well, if not better, than a human junior engineer. Sometimes it can't solve a bug, but human junior engineers get in the same situation too.
  • kubb 1y ago
    I believe that they work particularly well for CRUD in known frameworks like Rails. OTOH I tried building a native Windows Application using Direct2D in Rust and it was a disaster. I wish people could be a bit more open about what they build.
    • quantumHazer 1y ago
      yeah, tipically they are building a to do list and organizer app and have not found that github is flooded with college students' project of their revolutionary to-do apps
      • kubb 1y ago
        I don’t want to dismiss or disrespect anyone’s work. But I never see precise descriptions of categories of tasks that work well, it’s all based on vibes.
• otterley 1y ago
  On the contrary, Kiro (https://kiro.dev https://kiro.dev) is showing that it can be done by breaking down software engineering into multiple stages (requirements, design, and tasks) and then breaking the tasks down into discrete subtasks. Each of those can then be customized and refined as much as you like. It will even sketch out initial documents for all three. It’s still early days, but we are learning that as with software written exclusively by humans, the more specific the specifications are, the more likely the result will be as you intended.
  • quantumHazer 1y ago
    a 1 minutes research on the internet led me to discover that you are MARKETING MANAGER at amazon. so your take is full of conflict of interest and this should be disclosed.
    • dmacfour 1y ago
      There's an absurd amount of astroturfing in discussions about AI. Especially on Reddit.
    • otterley 1y ago
      Fair enough and I apologize for not disclosing it. However, Kiro is not a service in scope for me, and this is my own opinion, not that of the company. And it’s not a conflict of interest. I’m free to criticize my company if I like.
• anotheryou 1y ago
  Maybe we should let it build a mental model in documentation markdown files? Vibing I often let it explain the implemented business logic (instead of reading the code directly) and judge that.
• guluarte 1y ago
  Turns out, English is pretty bad for creating deterministic software. If you are vibe coding, you either are happy with the randomness generated by the LLMs or you enter a loop to try to generate a deterministic output, in which case using a programming language could have been easier.
  • tempodox 1y ago
    That's what I don't understand about AI coding fans. Instead of using a language that was designed to produce executable code, they insert another translation stage with a much murkier and fuzzier language. So you have to learn a completely new interface that is less fit for the task for the benefit of uncertain outcomes. And woe betide you if you step outside the most mainstream of mainstreams, where there's not an overabundance of training data.
    • bagacrap 1y ago
      It's because the AI coding fans already don't know programming languages, so they're learning a new language/interface either way. That, and their software doesn't actually have any users, I find.
• deleted 1y ago
  [deleted]
• lysecret 1y ago
  Bit of a click baity title since thy can definitely help in building software. However, I agree with the main thesis (that they can’t do it on their own). Also related to this this whole idea of “we will easily fix memory next” will turn out to be the same as “we can fix vision in one summer” turned out it’s 30 years later, much improved but still not fixed. Memory is hard.
• robomartin 1y ago
  I decided to jump into the deep end of the pool and complete two projects using Cursor with it's default AI setup. The first project is a C++ embedded device. The second is a sophisticated Django-based UI front end for a hardware device (so, python interacting with hardware and various JS libraries handling most of the front end). So far I am deeper into the Django project than the C++ embedded project. It's interesting. I had already hand-coded a conceptual version of the UI just to play with UI and interaction ideas. I handed this to Cursor as well as a very detailed specification for the entire project, including directory structure, libraries, where to use what and why, etc. In other words, exactly what I would provide a contractor or company if I were to outsource this project. I also told it to take a first stab at the front end based on the hand-coded version I plopped into a temporary project directory. And then I channeled Jean-Luc Picard and said "Engage!". The first iteration took a few minutes. It was surprisingly functional and complete. Yet, of course, it had problems. For example, it failed to separate various screens into separate independent Django apps. It failed to separate the one big beautiful CSS and JS files into independent app-specific CSS and JS files. In general, it ignored separation of concerns and just made it all work. This is the kind of thing you might expect from a junior programmer/fresh grad. Achieving separation of concerns and other undesirable cross-pollination of code took some effort. LLM's don't really understand. They simulate understanding very well, but, at the end of the day, I don't think we are there. They tend to get stuck and make dumb mistakes. The process to get to something that is now close to a release candidate entailed an interesting combination of manual editing and "molding" of the code base with short, precise and scope-limited instructions for Cursor. For my workflow I am finding that limiting what I ask AI to do delivers better results. Go too wide and it can be in a range between unpredictable and frustrating. Speaking of frustrations, one of the most mind-numbing things it does every so often is also in a range, between completely destroying prior work or selectively eliminating or modifying functionality that used to work. This is why limiting the scope, for me, has been a much better path. If I tell it to do something in app A, there's a reasonable probability that it isn't going to mess with and damage the work done in app B. This issue means that testing become far more important in this workflow, because, on every iteration, you have no idea what functionality may have been altered or damaged. It will also go nuts and do things you never asked it to do. For example, I was in the process of redoing the UI for one of the apps. For some reason it decided it was a good idea to change the UI for one of the other apps, remove all controls and replace them with controls it thought were appropriate or relevant (which wasn't even remotely the case). And, no, I did not ask it to touch anything other than the app we were working on. Note: For those not familiar with Django, think of an app as a page with mostly self-contained functionality. Apps (pages) can share data with each other through various means, but, for the most part, the idea is that they are designed as independent units that can be plucked out of a project and plugged into another (in theory). The other thing I've been doing is using ChatGPT and Cursor simultaneously. While Cursor is working I work with ChatGPT on the browser to plan the next steps, evaluate options (libraries, implementation, etc.) and even create quick stand-alone single file HTML tests I can run without having to plug into the Django project to test ideas. I like this very much. It works well for me. It allows me to explore ideas and options in the context of an OpenAI project and test things without the potential to confuse Cursor. I have been trying to limit Cursor to being a programmer, rather than having long exploratory conversations. Based on this experience, one thing is very clear to me: If you don't know what you are doing, you are screwed. While the OpenAI demo where they have v5 develop a French language teaching app is cool and great, I cannot see people who don't know how to code producing anything that would be safe to bet the farm on. The code can be great and it can also be horrific. It can be well designed and it can be something that would cause you to fail your final exams in a software engineering course. There's great variability and you have to get your hands in there, understand and edit code by hand as part of the process. Overall, I do like what I am seeing. Anyone who has done non-trivial projects in Django knows that there's a lot of busy boilerplate typing that is just a pain in the ass. With Cursor, that evaporates and you can focus on where the real value lies: The problem you are trying to solve. I jump into the embedded C++ project next week. I've already done some of it, but I'm in that mental space 100% next week. Looking forward to new discoveries. The other reality is simple: This is the worse this will ever be. And it is already pretty good.
• shad42 1y ago
  LLMs are powerful assistants—as long as the user keeps a firm mental model of the problem. That’s why, for now, they complement software engineers rather than replace them (at least today). When you already know exactly what needs to be built and simply want to skip the drudgery of boilerplate or repetitive tasks, a coding CLI is great: it handles the grunt work so you can stay focused on the high-level design and decision-making that truly matter (and also more fun).
• reramuyc 1y ago
  [dead]
• VladTepes2025 1y ago
  Have faith in AI, one day it will do what we hallucinate it can do!
• deleted 1y ago
  [deleted]
• mikewarot 1y ago
  3 months ago, I would have agreed with much of this article, however... In the past week, I watched this video[1] from Welch Labs about how deep networks work, and it inspired an idea. I spent some time "vibe coding" with Visual Studio Code's ChatGPT5 preview and had it generate a python framework that can take an image, and teach a small network how to generate that one sample image. The network was simple... 2 inputs (x,y), 3 outputs (r,g,b), and a number of hidden layers with a specified number of nodes per layer. It's an agent, it writes code, tests it, fixes problems, and it pretty much just works. As I explored the space of image generation, I had it add options over time, and it all just worked. Unlike previous efforts, I didn't have to copy/paste error messages in and try to figure out how things broke. I was pleasantly surprised that the code just worked in a manner close to what I wanted. The only real problem I had was getting .venv working right, and that's more of an install issue rather then the LLMs fault. I've got to say, I'm quite impressed with Python's argparse library. It's amazing how much detail you can get out of a 4 hidden layers of 64 values, and 3 output channels (rgb), if you're willing to through a few days of CPU time at it. My goal is to see just how small of a network I can make to generate my favorite photo. As it iterates through checkpoints, I have it output an image with the current values, to compare against the original, it's quite fascinating to watch as it folds the latent space to capture major features of the photo, then folds some more to catch smaller details, over and over, as the signal to noise ratio very slowly increases over the hours. As for ChatGPT5, maybe I just haven't run out of context window yet, but for now, it all just seems like magic. [1] https://www.youtube.com/watch?v=qx7hirqgfuU https://www.youtube.com/watch?v=qx7hirqgfuU
• random2019 1y ago
  [dead]
• crabmusket 1y ago
  The 4 step process outlined at the start of this article really reminds me of Deutsch's The Beginning of Infinity: > The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. (This is rephrased Karl Popper, and Popper cites an intellectual lineage beginning somewhere around Parmenides.) I see writing tests as a criticism of the code you wrote, which itself was a conjecture. Both are attempting to approach an explanation in your mind, some platonic idea that you think you are putting on paper. The code is an attempt to do so, the test is criticism from a different direction that you have done so.
  • sovietswag 1y ago
    That quote about conjecture reminds me of a big point from Zen and the Art of Motorcycle Maintenance. The author suggests that 'science' / 'the scientific method' don't actually account for the process by which ideas/hypotheses come into existence, science only comes into play once the hypothesis appears (from whence does it appear?). He calls that magic smoke 'Quality'. (Using the language you cited, I guess we would be asking about where the conjecture itself comes from). I'm realizing now that this is tangential to your point, sorry, but thanks for posting this interesting comment.
    • crabmusket 1y ago
      It's an interesting tangent! I agree with that idea, what we are taught as the "scientific method" does kind of assume you've already had the idea. The method helps you test or criticise it. I don't remember if Deutsch has an opinion on where conjecture comes from, but I think it's bound up with creativity... "quality" sounds like an interesting way of putting it.
  • card_zero 1y ago
    David Deutsch on LLMs: >LLMs are trained to imitate patterns of language, not to discover or verify truth. So, when asked to speak as an expert in an area where perceived experts have a widespread misconception, the LLM will parrot that misconception, adopting the register and vocabulary of experts. >This is dangerous when people take that register and vocabulary as conferring authority, or the very nature of an LLM as conferring truth or impartiality, which it cannot. They then amplify and entrench the misconception. Only distantly related to the subject, but it's in the general area of mental models. I suppose if you're trying to code a cliche, the LLM will help you get it right, and if you're trying to create something original, the LLM will help you get it wrong.
• madrasman 1y ago
  The 2 iOS apps that I published (mid level complexity and work well) say otherwise. I was blown away by what cursor + o3 could do.
• lisbbb 1y ago
  Back around, I don't even know, 2013? A colleague and I were working on updating a system that scanned in letters with mail order forms. The workers would lay the items from the envelopes in order on a conveyor type scanner. They had to lay them down in order: order form, payment check, envelope. The system would scan each document and add two blank fake scanned pages after each envelope. The company that set it up billed by scanned page. We figured out that you didn't need the blank pages as a delimiter because the envelope could reliably serve as that. By the way, the OCR was so bad that they never got the order forms to scan automatically, but people had to examine the order form as a pdf doc and key in everything instead. By eliminating the fake, nonsensical blank scanned pages, we saved the company over $1M/year in costs. We never got a single accolade or pat on the back or anything for that. Can AI do that, though?
• bubblyworld 1y ago
  I think the argument here is nonsense. LLMs clearly work differently to human cognition, so pointing out a difference between how LLMs and humans approach a problem and calling that the reason that they can't build software makes no sense. Plausibly there are many ways to build software that don't make sense to a human. That said, I agree with the conclusion. They do seem to be missing coherent models of what they work on - perhaps part of the reason they do so poorly on benchmarks like ARC, which are designed to elicit that kind of skill?
• nicbou 1y ago
  I have never tried seriously coding with AI. I just ask ChatGPT for snippets that I can verify, to save a few round trips to Google and API docs. However the other day I gave ChatGPT a relatively simple assignment, and it kept ignoring the rules. Every time I corrected it, it broke a different rule. I was asking it for gender-neutral names, but it kept giving last names like Orlov (which becomes Orlova), or first names that are purely masculine. Is it the same with vibe coding?
  • parimal7 1y ago
    Has been my experience as well, never used ChatGPT for anything but as an interface to external documentations and understanding unfamiliar syntax. Tried using it for the first time for vibe coding and was quite disappointed with the overall result, felt like a college student hastily copy pasting code from different sources for a project due tomorrow. Maybe I just gave bad prompts…
  • energy123 1y ago
    It's the same. Every assumption, taste, doc, opinion, edge case, unstated knowledge, requirement, exception to the requirement, intention, should be provided for best results. It's not crucial for throwaway weekend projects, but for harder things, it is. I find it to be the most challenging part. There's a large amount of unstated assumptions that you take for granted, and if you don't provide them all upfront, you'll need to regenerate the code, again and again. I now invest a lot of time into writing all this down before I generate any code.
• kachapopopow 1y ago
  I'm tired of people telling me that llms are bad at building software without trying to sit down, learn how to properly use claude code, when to use it and learn when you shouldn't use it. Cursor is a joke tho, windsurf is pretty okay.
  • ale 1y ago
    This is not a you're-holding-the-llm-wrong problem though, AI tools are simply not capable of creating mental models for problem solving. Sounds like you're tired of hearing that LLMs are not a silver bullet.
    • kachapopopow 1y ago
      Exactly, there's things you aren't shouldn't do with an llm. But generating helm charts, configs, action workflows, building specs and then implementing based on them? Simply a no-brainer.
  • TrackerFF 1y ago
    LLMs are as good as the inputs a person gives them. Right now the scene is very polarized. You have the "AI is a failure, you can build anything serious, this bubble is going to pop any day now" camp, and the "AI has revolutionized my workflow, I am now 10x more productive" camp. I mean these types of posts blow up here every. single. day.
    • kachapopopow 1y ago
      It's neither useless nor it's 10x. It's a solid 1.2x - 1.8x.
• _pdp_ 1y ago
  LLMs cannot build software on their own yet. They are can sure build software with some help.
• siva7 1y ago
  > But, we firmly believe that (at least for now) you are in the drivers seat, and the LLM is just another tool to reach for. So does Microsoft and Github. At least that's what they were telling us the whole time. Oh wait.. they changed their mind i think a week ago.
• morteify 1y ago
  [dead]
• kypro 1y ago
  > LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. That's actually an interesting point, and something I've noticed a lot myself. I find LLMs are very good at hacking around test failures, but unless the test is failing for a trivial reason often it's pointing at some more fundamental issue with the underlying logic of the application which LLMs don't seem to be able to pick up on, likely because they don't have a comprehensive mental model of how the system should work. I don't want to point fingers, but I've been seeing this quite a bit in the code of colleagues who heavily use LLMs. On the surface the code looks fine, and they've produced tests which pass, but when you think about it for more than a minute you realise it doesn't really capture nuance of the requirements, and in a way a human who had a mental model of the how the system probably wouldn't have done... Sometimes humans miss things in the logic when they're writing code, but these look more like mistakes in a line rather than a fundamental failure to comprehend and model the problem. And I know this isn't the case, because when you talk to these developers they get the problem perfectly well. To know when the code needs fixing or a test you need a very clear idea of what should be happening and LLMs just don't. I don't know why that is. Maybe it's just they're missing context from the hours of reading tickets and technical discussions, or maybe it's their failure to ask questions when they're unsure of what should be happening. I don't know if this a fundamental limitation of LLMs (I'd suspect not personally), but this is a problem when using LLMs to code today and one that more compute alone probably can't fix.
• kleyd 1y ago
  Current LLMs look a lot like a very advanced 'old brain' to me. While context engineering looks like optimizing the working memory. What's missing is a part with more plasticity that can work in parallel and bi-directionally interact with the current static models in real-time. This would mean individually trained models based on their experience so that knowledge is not translated to context, but to weight adjustments.
  • movpasd 1y ago
    That's also my view. It's clear that these models are more than pure language algorithms. Somewhere within the hidden layers are real, effective working models of how the world works. But the power of real humans is the ability to learn on-the-fly. Disclaimer: These are my not-terribly-informed layperson's thoughts :^) The attention mechanism does seem to give us a certain adaptability (especially in the context of research showing chain-of-thought "hidden reasoning") but I'm not sure that it's enough. Thing is, earlier language models used recurrent units that would be able to store intermediate data, which would give more of a foothold for these kind of on-the-fly adjustments. And here is where the theory hits the brick wall of engineering. Transformers are not just a pure machine learning innovation, the key is that they are massively scalable, and my understand is part of this comes from the _lack_ of recurrence. I guess this is where the interest in foundation models comes from. If you could take a codebase as a whole and turn it into effective training data to adjust the weights of an existing, more broadly-trained model, But is this possible with a single codebase's worth of data? Here again we see the power of human intelligence at work: the ability to quite consciously develop new mental models even given very little data. I imagine this is made possible by leaning on very general internal world-models that let us predict the outcomes of even quite complex unseen ("out-of-distribution") situations, and that gives us extra data. It's what we experience as the frustrations and difficulties of the learning process.
• thrance 1y ago
  I had an awful, terrible experience with GPT5 a few days ago, that made me remember why I don't use LLMs, and renewed my promise to not use them for at least a year more. I am a relative newbie to GPU development, and was writing a simple 2D renderer with WebGPU and its rust implementation, wgpu. The goal is to draw a few textures to a buffer, and then draw that buffer to the screen with a CRT effect applied. I got 99% of the way there on my own, reading the guide, but then got stumped on a runtime error message. Something like "Texture was destroyed while its semaphore wasn't released". Looking around my code, I see no textures ever being released. I decide to give the LLM a go, and ask it to help me, and it very enthusiastically gives a few thing to try. I try them, nothing works. It corrects itself with more things to try, more modifications to my code. Each time giving a plausible explanation as to what went wrong. Each time extra confident that it got the issue pinned down this time. After maybe two very frustrating hours, I tell it to go fuck itself, close the tab and switch my brain on again. 10 minutes later, I notice my buffer's format doesn't match the one used in the render pass that draws to it. Correct that, compile, and it works. I genuinely don't understand what those pro-LLM-coding guys are doing that they find AIs helpful. I can manage the easy parts of my job on my own, and it fails miserably on the hard parts. Are those people only writing boilerplate all day long?
  • spopejoy 1y ago
    It's more that if there isn't 10 stack overflows on your exact problem then you're out of luck. LLMs are 100% useless for: - non-mainstream languages - langs without massive corpuses of online tutorials and SOs I was going to add "langs without large online open-source codebases" but if they don't have extensive, reliable SOs (haskell, Solidity, even rust) then LLMs struggle to the point of uselessness, because LLMs don't actually trawl through random codebases and magically turn it into cookbooks and tutorials. Indeed, an emergent problem with LLMs is they are going to recreate the 90s/2000s PL dark ages where "never write a new language" was mantra. Now it will be because a new lang can never hope to get sufficient LLM training data. EDIT to your problem, it isn't the lang but the domain. If experts aren't dumping huge volumes of explanatory text online, an LLM can't help you even in a mainstream lang.
• _joel 1y ago
  Side comment, I love the typography of the site. Easy to read.
• arendtio 1y ago
  This is the post that the people at Anthropic and Cursor should read. > But what they cannot do is maintain clear mental models. The emphasis should be on maintain. At some point, the AI tends to develop a mental model, but over time, it changes in unexpected ways or becomes absent altogether. In addition, the quality of the mental models is often not that good to begin with.
• _1tem 1y ago
  This was Peter Naur’s observation in his 1985 paper Programming as Theory Building. One of the best papers I’ve ever read. Programming IS mostly building castles in the air of your head, not the code on the screen. If you do the thinking and let the LLM do the typing it works incredibly well. I can write code 10x faster with AI, but I’m maintaining the mental model in my head, the “theory” as Naur calls it. But if you try to outsource the theory to the LLM (build me an app that does X) you’re bound to fail in horrible ways. That’s why Claude Code is amazing but Replit can only do basic toy apps.
• tw1984 1y ago
  the mentioned "two similar mental models" is an interesting of looking at the problem. if that is the actual case, seems to me that a much better model plus an smart enough agent should be able to largely solve the problem. interesting time, interesting issue.
• tjtryon 1y ago
  I find that if you are great at pseudocode, and great at defining a programs logic in the pseudocode, along with defining all cases possible, and trying to add user id10t error issue to the logic, you can get a pretty good framework of a program to start to work with. I also ask for comments for all logic, loops, if statements and functions that are instructional, yet easy enough that a 5th grader could understand. This framework usually comes as a version .8 for further development manually or even .9 beta test/debugging level. Occasionally, I've seen version 1.0 release candidate 1 level work, where I need to try to verify the functionality, AND try to find ways that users may try that will break functionality. Whichever version I end up with, it still involves manual coding, there is no escaping that. Using a LLM just saves (a lot) of time with the initial framework and program logic framework.
• anonu 1y ago
  I think we just need better AI coding tools: the space is still a blue ocean for the ones that can figure it out. The article highlights the key problem with AI tools today - which is that there doesn't seem to be a high level planning step (aka a mental model) to start with. Every time you ask a new question, the LLM starts from scratch.
• gwbas1c 1y ago
  LLMs can't reason: They're a statistical model that can only copy. In many cases though, the copy that the LLM generates is either "good enough" or a great thing to start with.
• germandiago 1y ago
  This very afternoon I coded with ChatGPT what would have taken me maybe some more time. Maybe 30-50% more. True that it was more-or-less independent code. I think these tools are very effective at four things: 1. initial scaffolding (and isolated scripts also) 2. reviewing code and finding insights of where an error that we humans are very bad at spotting could be, given a description and an area of code. 3. If you are not an expert in something (for me that is frontend) it helps. 4. finding ideas on how to approach a problem through a conversation But if you are well-versed, you still have more context and better decision making. They are ok-ish but not expert level. It also comes with downsides if you abuse them: you could end up not understanding your codebases well enough or adding bad quality code that seems to work. Also, if you initially think you can solve a problem with AI and it leads you the bad way, you end up wasting more time than what you save. All in all, I find them good at scaffolding, asking ideas or solutions and spotting potential bugs. As a whole they are accelerators but not replacements, IMHO.
• kashnote 1y ago
  I actually don't have doubts that LLMs are quite good at writing software. The problem for me is one of practicality. If, after hundreds of lines of AI-written code, I noticed some sort of issue (regarding scale, security, formatting, logic, etc.), I'm basically forced to start over. We all know that reading code is way less pleasant than writing code. So, for me, LLMs can be very useful for writing code that I know is going to be correct without having to go back through it. For example, basic TRPC CRUD functions.
• kpil 1y ago
  This is underrated and applies to almost all roles involved in product development. > the distinguishing factor of effective engineers is their ability to build and maintain clear mental models.
• alphazard 1y ago
  Has anyone been able to separate creativity from hallucination in LLMs? As in: we can affect one without the other, or measure one without the other. In humans we understand when someone is doing one vs. the other. The biggest difference is that a creative person knows they will have to take action to bring about their vision, but a hallucinating person thinks their vision already exists. It seems like the LLM phenomenon that we call hallucination is descriptively the same phenomenon that we call creativity in other contexts. If the LLM adds a new function or feature to the current project as part of work on another feature, that's creativity. But if it assumes a function or type in another project, which can't easily be changed, we call that hallucination. Even though it could just as easily add that feature if it had access to that code as well.
  • L_226 1y ago
    Has anyone been able to separate creativity from hallucination in humans?
    • raincole 1y ago
      Yes. For example, this commenter thinks slamming "what about humans" sticker onto any criticism or discussion about LLM is some kind of creative comeback. However it really isn't.
• seemaze 1y ago
  Zed hasn't won me over from Vim (Neovim) yet.., but I will say I appreciate the keyboard navigation built into the website! KYC :)
• Daniel-Gara 1y ago
  Good diagnosis of the loop. But “LLMs can’t build software” is mostly a statement about interface, not capability. If we ask a stochastic parrot to behave like a staff engineer in its head, it fails. If we reshape the work so the mental model lives outside the model—executable specs, tight tests, ADRs as shared state, small DSLs, and an orchestrator that forces evidence before code—agents can move through the same loop on bounded problems. In other words, they don’t need an internal model if the environment gives them one. The better question isn’t “can an LLM maintain two mental models?” but “how much of this problem can we make machine-checkable?” Where we can’t (socio-technical trade-offs, ambiguous requirements), a human owns the decisions. Where we can (migrations, glue, refactors guarded by tests), the agent owns the keystrokes. Today’s failure modes (omission, recency bias, hallucination) are real, but mitigated by durable memory, runbooks, and mandated check-ins the tool can’t skip. So: not “can’t build software”, but “can’t be the tech lead”. Yet.
• ughitsaaron 1y ago
  > When you watch someone who knows what they are doing, you'll see them looping over the following steps: > Build a mental model of the requirements > Write code that (hopefully?!) does that > Build a mental model of what the code actually does > Identify the differences, and update the code (or the requirements). This is pretty right on but I think it leaves out an aspect of writing code that I think is often pretty under appreciated. Code does two things at once: it provides a set of instructions to a machine and it communicates the authors' understanding of the program behavior those instructions are intended to express. I think this is a large part of what makes programming so fascinating and frustrating. It's what's behind the cliche that "naming things" is one of the hardest parts of programming. In growing software systems it's often not enough that a feature's implementation works. Ideally, that implementation should impose a minimum barrier to understanding for contributors to do something with it afterward. I'm not convinced this is an aspect of software development that LLMs will be able to meaningfully achieve.
• AtNightWeCode 1y ago
  I typically design solutions to minimize the cognitive load on coders boarding my projects. They should not need a mental model to add features. Same with AI.
• Anamon 1y ago
  > they are excellent at synthesizing requirements and documentation Hard disagree. I'm sure you think it's excellent if you're the person who was supposed to be writing it, but your opinion is influenced by your happiness that you didn't have to do it. LLMs are good at generating the kind of documentation that shouldn't be written in the first place, because it's just a wordier rephrasing of what's already in the code, and therefore wastes the time of anyone who reads it. LLMs suck at documentation for the same reasons the author identifies as the reasons why they suck at programming. The additional value I'm looking for when reading documentation is mostly the mental models.
• terminatornet 1y ago
  Every blog post about AI now is something along the lines of "hmm, yeah I guess AI doesn't really replace software engineers like they said it would, but maybe it'll be better tomorrow ¯\_(ツ)_/¯"
• dingi 1y ago
  "AI" is becoming meaningless. Most LLMs aren't intelligent; they're just sophisticated pattern-matchers. I call them AI-Like, inspired by "Unix-like." An AI-Like system produces outputs that mimic intelligence, has no real understanding or reasoning, and works only in narrow contexts. It’s a small shift in wording, but it sets expectations straight. Humans are sophisticated pattern-matchers with consciousness, goals, and self-reflection. LLMs are just pattern-matchers. Hence: AI-Like, not AI.