7 ms·
Measuring the sloppiness of code
- tphyahoo2 21d ago"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate" I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
- fosterfriends 21d agoI love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
- loveparade 21d agoIf you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
- yehoshuapw 21d agothat is true, but if the metric is what we want optimized, then that's fine. However it is more likely to be something which can be detached..
- datsci_est_2015 21d agoCode quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.
- peder 21d agoSooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
- alecbz 21d ago> that's primarily a human concern that's unrelated to the final output. It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
- stiiv 21d agoI think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactoring-economic-benefit.html https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree. On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
- FuckButtons 21d agoI don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
- siscia 21d agoIt is not clear to me how the verbosity metrics works. Can someone shades more light on it?
- yorwba 21d agoIt's a bunch of rules flagging specific patterns that could be expressed more succinctly: https://github.com/gabeorlanski/scb-check/tree/a8618228939def726c2ec48b354693e5aa1999d5/src/scb_check/resources/slop_rules https://github.com/gabeorlanski/scb-check/tree/a8618228939de...
- mococa 21d agoWell... Coding was never the most part of work I spent time.
- marginalia_nu 21d agoCongrats on your promotion to full time jira management!
- softwaredoug 21d agoCoding might be "solved" but coding still is the best way to build your own mental model of the solution space. Which is more important to you: Velocity to a solution? Or velocity to understanding?
- nucleative 21d agoInteresting way to lay it out. For us understanding is obviously crucial for prod and repeatable business functions. Velocity to solution is default for almost everyone else, especially one-off or low impact / low consequence of failure projects.
- timbaboon 21d agoVelocity to solution is default for senior management, that's for sure ;)
- hax0ron3 21d agoI code to make money, and the kind of stuff I work on doesn't kill people or lose massive amounts of money if it has bugs, so to me velocity to a solution is much more important than velocity to understanding.
- greenowl 20d agoVelocity to a paycheck
- vanschelven 21d ago> I was disappointed at how “vibes based” the industry seems at the moment. Alan Kay called programming "Pop Culture" some 20 years ago[0] [0] https://queue.acm.org/doi/10.1145/1039511.1039523 https://queue.acm.org/doi/10.1145/1039511.1039523
- scronkfinkle 21d agoThere is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
- victorbjorklund 21d agoIndeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”
- marginalia_nu 21d agoTo be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.
- esafak 21d agoAnd it can -- if an able person wields it.
- victorbjorklund 21d agoYea, that extreme side exists too. Truth is inbetween. AI with the instructions from a dev that knows what it is doing writes better code than most regular 9-17 devs.
- shimman 21d agoThis was always due to pressures by management and the company environment, not the workers themselves. It's hard to blame the people writing code when they have to deal with nontechnical leadership that wants to have a feature factory or never given appropriate resources to solve problems. Blaming workers is always an excuse by poor management.
- Retric 21d ago
- Varelion 21d agoCoding is solved, but AI companies are still hiring software engineers?
- elcritch 21d agoBest explanation I’ve heard is that “coding is solved but software engineering hasn’t”.
- deleted 21d ago[deleted]
- devld 21d ago* slop engineering
- Dlemlo 21d agoSo they still might do but in parallel the software engineering market in china and india is collapsing.
- Varelion 21d agoSource?
- Dlemlo 21d agoLots of chatter online, i have seen a few videos on this tpic on YT. https://www.hays-china.cn/en/press-release/content/%E4%B8%AD%E5%9B%BD%E5%A4%A7%E9%99%86%E5%B8%82%E5%9C%BA%E7%A7%91%E6%8A%80%E5%B2%97%E4%BD%8D%E5%8F%98%E5%8C%96-%E9%9A%8F%E7%9D%80ai%E8%BF%90%E7%94%A8%E5%AF%B9%E9%9C%80%E6%B1%82%E4%BE%A7%E7%9A%84%E5%BD%B1%E5%93%8D%EF%BC%8C%E8%BD%AF%E4%BB%B6%E5%BC%80%E5%8F%91#:~:text=Organisations%20are%20encouraging%20the%20use%20of%20AI,Wei%2C%20Managing%20Director%20at%20Hays%20Greater%20China. https://www.hays-china.cn/en/press-release/content/%E4%B8%AD...
- guywithahat 21d agoI think he's referring to contracting companies. Cheap software used to be outsourced to India, but it's now getting outsourced to AI. Companies that produce novel things are still hiring, often to manage agents and review output.
- linsomniac 21d agoI wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.
- _pdp_ 21d agoIt will be solved when there is no more code left to write. Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before. Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task. I might be wrong :)
- wang_li 21d agoCode and software and applications are an intermediate stage. The final stage is an AI/LLM that just does the thing that is needed without any code being written, there are no applications or programs, just an AI that does everything.
- cocoto 21d agoThis doesn’t scale at all for some problems where a specific algorithm is needed (graph problems for instance).
- bluecheese452 21d agoClaude do the needful.
- datsci_est_2015 21d agoSo how will separate systems communicate with each other? Or are there no separate systems? That’s the digital singularity I suppose.
- _pdp_ 20d agoAI needs tools as much as humans do.
- Dlemlo 21d agoI'm pretty sure we reached peak software developer jobs due to ai. The market is already collapsing in China and India.
- danbruc 21d agoCoding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.
- zsoltkacsandi 21d agoThat is very well put and summarizes what distinguishes real software development from vibe coding.
- Dlemlo 21d agoI have seen so much production code with garbage code and massive bugs, the industry doesn't care for 'real'.
- danbruc 21d agoThey care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.
- Dlemlo 21d agoNot even in principle. Like how often you had to fight a product manager to do this or that. "Does that code work?" "yes" "so lets ship it" "but its not good" "but it works right?"
- danbruc 21d agoBut they also ask why the next feature is taking so long, because of the mess we created with the last feature, and I would consider this caring in principle. Unfortunately this almost always filed as unavoidable.
- deleted 21d ago[deleted]
- Kinrany 21d agoNumber of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy. The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can. One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
- cjalmeida 21d ago>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure. This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
- gpugreg 21d agoI've had some success with tokens as a measure of complexity instead of number of lines, but should be combined with additional rules, e.g. disallowing lambdas, exec, eval, compile, __import__ and complex list comprehensions for Python. Fortunately, Python's "ast" module makes this quite easy.
- Xenoamorphous 21d agoSad as it might sound, I think we might have to stop worrying about the code.
- guayusa 21d agoSolving consciousness ;)
- cheney_2004 21d agoYa, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
- avgDev 21d agoI encountered bugs created by AI. No matter how many times it tried it could not fix the bug, it was introducing so much slop to work around the issue. I finally gave up and read documentation for 15 minutes and solved the problem. I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
- NichoPaolucci 20d agoI liken it to contractors or overseas engineers. They don't really care about the long-term effects of their code. I does the thing, adds some code, adds test that cover that code, and responds with "Done." At a fully agentic "No looking at code" company, we now don't have a great understanding of that code, we don't know whether it's a good implementation or not, whether it fits our current patterns, is maintainable, or composable. We're entirely in the dark, at the behest of an agent at this point - if we want to maintain velocity. (And I would love to know whether that velocity is real or just perceived). It feels like the code part is faster? But also feels like I have to spend a lot more time up front working through a problem to understand it. In the past, I gained that understanding WHILE I was working through the code. Who knows.
- tloopff 21d agoClaim: AI writes almost perfect code. Reality: earandil.com uses 170% CPU in Firefox. What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"? Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
- antoni4040 21d agoCoding has been solved for 20 years at least. 90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM. 90% of code before LLMs was badly copied from StackOverflow anyway. That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that. The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit. (sorry if I sound incoherent, just some thoughts while I'm commuting)
- Pannoniae 20d ago>LLMs haven't created a I don't think this is true. They very well might have, but a specific one, not a generic reusable version you find on GitHub or whatever.
- j45 21d agoIt will be fun to try and deterministically define sloppiness relative to a tool that is not deterministic. Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.
- justinmarsan 21d agoHaving reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff... It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling... But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
- datlife 21d agocurious how did you set up your Architecture subagen?
- bobkb 21d agoCoding just a stage in the software development. Design and specifications which can help in coding is not solved at all and may never - the end result is software reliability is not a solved problem.
- plourens 21d agoSounds right to me. As AI has picked up, I have found myself drawn to resources that (I hope) help improve my taste and judgment not only of code but also of overall software design. The lessons in Code Complete, for example, have been helpful to me in thinking about managing the complexity of a codebase. It was useful before, too, but perhaps even more so now.
- conqrr 21d agoCoding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team. If humans increasingly are kept out of coding, then who holds the mental model? If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions. So is coding solved now? it was already solved decades ago.
- kolinko 21d agoWho says anyone needs to hold a mental model? Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase. Before LLMs we couldn’t have built tools that would deliver such views, now we can.
- NichoPaolucci 21d agoI mean... Nobody NEEDs to hold a mental model, but we've been increasingly trying to build that mental model into an agentic codebase... maybe we're doing it wrong but it is NOT going well. The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide. We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects. Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
- FiberBundle 21d agoDoes anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
- gritzko 21d agoThat kind of complexity is combinatorial so "a 4x4 doesn't stop you getting stuck, it just means you get stuck further from help."
- kolinko 21d agofirst you’d need to define how to define/measure the complexity of code when it comes to this case.
- NichoPaolucci 21d agoIn theory - if an LLM could handle infinite complexity, I still think that the business issues + decisions end up getting in the way somewhere. AI: "You asked to add feature X. Here are 25 questions that impact feature Z, B, and C in your gigantic codebase" Developer: I can answer 8 of these questions... Guess I need to go figure out the rest of them. Writing the code + building the functionality has always been the easy part.
- pdimitar 20d agoYeah, exactly my experience. Especially ever since Fable came out, I felt paralysed, many times, about how many decisions I have to hand it. Which is 10x harder when you handed it the reins to build the PR and you practically only knew the compressed initial problem statement. Then it stands up a draft PR and says: "Decisions you owe me: [insert bulleted list with 8 items each a paragraph long]" and you are like: "...oh shit". It becomes a very heavy and difficult exercise of it walking you through the implementation and the judgment calls it had to make along the way. VERY exhausting. There's a silver lining though: you do get to gradually clarify a proto project spec and various requirements, but boy does it take time and energy to re-contextualise when the bot tells you that you should make decisions.
- CuriouslyC 21d agoI started Valknut (https://github.com/sibyllinesoft/valknut https://github.com/sibyllinesoft/valknut) when I saw the writing on the wall regarding Agent code structure/abstractions/etc being a limiting factor in the ability to autonomously build projects. My experience was that good linters helped, but it wasn't enough, you needed to be able to enforce information-theoretic related organizing principles in addition to file/function LOC and local complexity metrics to guide agents on how to structure code. Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them. Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
- RickJWagner 21d agoIf coding is ever solved, and if software does it, sloppiness probably won’t matter much. Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one. Test suites will be important.
- technoplato 21d agoPlopping in my email to the author below in case anyone else is interested in this kind of thing: > There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions". In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc. Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows: L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on. Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against. I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission. I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition. Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me. Best, Michael PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20 https://x.com/technoplato/status/2090902061437030777?s=20
- neptvn 21d agoIt's a nice article that basically (rage)baits the readers before they realize the author actually disagrees with the premise of coding being solved. So here goes, my higher-level rant on "solving" something with AI. I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve". Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving. Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems. To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
- anilgulecha 21d ago> Qwen2.5-Coder-3B Basing it's findings of LLM as judge on this model, and then proceeding to ignore it. This article can be safely ignored as well. LLM as judge in harness evals is the way to go, for any of your custom needs. Design the eval well.
- LaffertyDev 21d agoI don't know where you pulled that from, its not in the article.
- anilgulecha 21d agoFollow the paper from the LLM as judge section.
- dherman 21d agoReally glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start! My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring. I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
- siscia 20d agoDo you have specific examples of what you mean here?
- toddwprice 21d agoCoding is solved, perhaps, with unlimited token spend on a frontier model. It remains to be seen if it that is prohibitively expensive forever. At my company, we token maxed while the getting was good. But when we had to switch to Anthropic's enterprise plan, and start paying per token, the shit really hit the fan. Now we're retreating back to sane cost levels and finding that - guess what? - people power might just be more cost effective. AI of course is an immense tool to leverage, but still too expensive to create loops and let it run. This will change over time of course, but assuming it is a solved problem is nonsense. Maybe if we solve cold fusion, yes. Until then, evolution is winning the war on entropy.
- NichoPaolucci 21d agoI'm interested in hearing more about your experience here. Were you all just full bore shipping a ton with the per token plans? Was it more effective? How many developers @ your company? It's really interesting to hear that some companies are back-pedaling. We "slowed" down a bit but are still very much using AI and intend to continue using it for... almost everything.
- greenowl 21d agoHow did the developers react when they had to go back to writing their own code instead of playing Nintendo Switch between prompting sessions?
- warkdarrior 20d ago> had to go back to writing their own code I hate it. My fingers are cramping and my head is hurting.
- suprjami 20d ago> Coding is solved, perhaps, with unlimited token spend on a frontier model. Coding is also a solved problem with unlimited salary budget on the very best developers.
- edf13 21d agoRecent PR I had to review... PR content: ``` Lots of AI slop.... .... .... Note: this will not build due to XYZ .... .... More AI slop .... .... End of PR ``` So the dev hadn't even read the PR comment himself and had blindly posted it!
- taosx 21d agoI have no idea where most people writing code have worked at but in all product and platform teams I worked at the code quality has been much higher than the latest slop SOTA llms can output. TLDR: coding is not solved. I have 2 projects, one it's a distributed platform, the other one is a general processing engine with an inner workflow engine; Since gpt 5.2 I've tried new models to work in these codebases where the code is of good quality and every time I gave the model a slice of work instead of a single step from that slice the code, the tests, the comments, the docs and everything else has been suboptimal, unmaintainable, complex, bloated and just slop, unless I micro-manage and do many passes. As a dev when you make a change you consider the broad picture, you consider the user, the codebase, future requirements, maintainability, performance, your team's understanding and some of these you do unconsciously. We are slow but that's for multiple good reasons, you push the organization/understanding forward not just loc of that specific project. I can't count how many PR notes or comments I've added considering teammates or just for a specific team member. I don't see any way forward for an LLM to reach that unless it reaches general problem solving, my definition of GAI that could tackle software development or "coding" would be a model that doesn't require additional pretraining to solve new tasks or improve how it solves tasks in the future, it would just learn as it's going. Can everything I mentioned be solved with current generation of LLMs and lot's of markdown and gates? Maybe... but the amount of effort required would be similar to the effort an expert system (pre-llm AI) would require to embed the rules, evolve them, check them everytime... which would require billions or trillions of tokens. --- off: I really like the discussions around how to prevent slop and bloated code as it's something it would benefit coding even without LLMs and can fit as another piece of automated infra for checking and ensuring code quality, I hope something materializes.
- oumua_don17 21d agoBefore AI, most of the code was already crap and slop. After AI, the volume of that crap and slop has increased exponentially.
- farhadhf 21d agoCoding is not solved. It's only solved when coding becomes something you do because you want to, just because you like doing it, the same way I bake bread at home because I like doing it, not because I have to. Right now we still have to be hands-on - to a lesser extent, yes - but we still have to review and hand holding AI agents to get things done.
- mikkelam 21d agoThis is once again RLHF loops. the AI labs are and have been 100% focused on correctness because it is easy to setup and validate. Adding one more function that almost does the same thing as another will not break anything. I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to set up environment to Train for this behavior though.
- brap 21d agoThere’s one thing I constantly see agents tripping over, I’m not sure what the right word for it would be, but it basically boils down to “making changes in the right places”. They seem to have very poor grasp of where things are supposed to be and they have a tendency to work against the existing architecture. Even in a world where agents are the only ones touching the code you can see how this ends poorly. Unlike correctness I’m not sure there’s an easy way to verify. I tried writing a few skills to encourage agents to spend time thinking about this but it doesn’t seem to generalize very well.
- vatsachak 20d agoMaintainability is hard because it requires much more planning than correctness. You need to think about how you can construct code which gives you the most amount of reusability while not compromising on readability and also thinking about future paths and how they are affected. It seems like solving the problem of constructing maintainable code is as hard as solving long scale planning in LLMs
- kloud 21d agoThe problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more. Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
- rstuart4133 20d agoHe used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code. The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.
- Rover222 21d agoI'm not shocked the top 2 comments are "coding is not solved" Pure cope, at least in most domains. The machines can do it better than the humans. I think most arguments against that are clinging onto old things that don't really matter any more, like coding style. If most of us are not coding by hand at all anymore, just directing and iterating with the agents, then coding is solved by a reasonable definition.
- hintymad 21d ago> LLMs are able to write almost perfectly correct code. This is kinda vague. Correct at what scale? I wonder if there's a measurement on the correctness per scale, and hopefully the scale is not just CLOC.
- dang 21d agoAll: please don't post generic reflexive reactions to titles. That's covered by this guideline, among others, in https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead." I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=reflective%20reflex%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
- rayiner 21d agoIs coding "solved?" I've been using Fable and Grok to vibecode a library to improve text extraction for legal PDFs. (Which have things like line numbers in transcripts that messes up text extraction.) This is a simple, dumb problem that should be like killing a mouse with a nuke. But I got to "pretty good" with a few iterations of not looking at the code or design. The program works, pretty well. But the design and heuristics are only 80% of the way there. Now, the models are just "churning" on the existing design, making ad hoc changes struggling to get to the last 20%. It seems like I'm going to have to turn my brain back on and think about what I actually want to do, which I was trying to avoid. It's better than writing assembly! But it still seems to be in the realm of "a new kind of tool" even for the actual coding part, without even caring about what the code looks like or security or anything like that.
- keeda 21d agoWithout knowing details of your approach, I would venture that your challenge is not with the coding per se but extracting structured data from PDFs. It’s a surprisingly hard problem because PDFs are optimized for preserving the visual structure and layout of the content for precise rendering and printing… NOT for preserving the logical structure of the data! Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it. A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.
- rayiner 21d agoThe use case is basically the same ones where a programmer would use grep on source files, and it’s too slow to run the model on it directly for the same reasons. Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.
- MichaelNolan 21d agoI was bit disappointed how few metrics the article mentioned. There are tons of code quality metrics that have been thought of over the last 40 years. We don’t have a good idea which ones are worth enforcing though. And we don’t know if the metrics that are good for humans are also good for LLMs https://dekobon.github.io/big-code-analysis/metrics.html https://dekobon.github.io/big-code-analysis/metrics.html https://dekobon.github.io/big-code-analysis/metrics-vcs.html https://dekobon.github.io/big-code-analysis/metrics-vcs.html
- drsopp 21d agoGoodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
- greenowl 20d ago> What is sloppy code anyway? Code that is overly and unnecessarily difficult and cumbersome for humans to work on. But given the way humans work on code nowadays doesn't involve actually writing it themselves, there's no such thing as sloppy code anymore. Hooray!
- bvrmn 20d ago> could we throw the definition into context and tell the LLM to avoid it? Every IMPORTANT RULE WHICH MUST BE FOLLOWED STRICTLY to not to do something from my AGENTS.md and skills are violated without any hesitation from LLM. Glorified autocomplete doesn't work this way. Positive rules are more stable on this regard.
- glouwbug 21d agoA good measure is communication. If there's a common understanding, then the origin doesn't necessarily matter.
- nilsherzig 20d agothe website of this company is using more resources than their main product is (would really appreciate a button to turn of the animation)
- fkozlowski 20d agoNice! It's interesting to see quantitative ways of measuring slop. I'm curious what _would_ happen if you did plug these back into the LLM as a form of feedback. Goodharting may happen... or it could get the LLM to generate cleaner code potentially?
- VCFundedGenYer 20d agoOnly frauds/vibecoders/non-programmers claim coding is "solved". This is nonsense.
- pbjerkeseth 20d agoI hopped into this hoping it would land where it did! I recently built in basic observations about cyclomatic complexity, churn, and authorship into an 'analysis' view of codebases in ouijit[1]. Some benefits can be reaped immediately (cyclomatic complexity), but others emerge over time (churn). A good example of this is say something like a 500 line file that has experienced 2500 lines of churn, and if that rate of churn is trending down or up. Surprisingly useful for understanding you've got a hotspot with an opportunity to pay down debt by spending more time on API design, or just breaking out whatever subset of the code is experiencing thrash. The funny thing about complexity is that assuming you're lintting/formatting well, you can do a poor mans check by just looking at something like average indentation per line, deepest line, etc. [1]: https://ouijit.com https://ouijit.com
- utopman 20d agoI hope this is not too provocative, but where I work the legacy code is such a spaghetti mess and so actively inefficient (queries in loops of loops and plenty of things like this) that just prompt nowaday llm to "write extensive and menignful tests for this module, then rewrite the module clean mvc small functions , small loc per files, well organized files then make pass tests again" often leads to insanely better ai "sloppy" resulting code, plus free test coverage where nothing existed, users instant satisfaction (software become suddenly responsive in less than a second). From this point it take not much more time to add proper ci/cd suite, rework database models and migrate things with now some comfidence. It works very well on small isolated projects and new builds where resulting tools are produced insanely faster than before and with not comparable average code quality (ai slop is 100x better here) from customer usual habits and expectations. It's because this approach raise significantly the quality level so quickly and so cheap that I advocate for it. It however does not work the same way for large legacy projects to revive. For larger projects the reciepe is almost the same but requires significantly more manual testing on each pass and carefull regression check. But it does work. Customer is happy, they take back "more" control over their legacy horrors at the cost of some extra loc but properly working code. There is no point to fight against "a bit too much code" in my case I don't want to speak for those that work in elite very sensitive human responsibilities software domains, but in a lot of place this is a revolution in my opinion.
- keeda 20d agoThe conclusion is not surprising really, because fundamentally how do you even quantify sloppiness, a famously broad and subjective characterization? I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality. People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality." An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures. But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs! Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow. So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
- ls-a 20d ago[dead]
- unsubscribetool 18d ago[dead]
- rolisz 20d agoI just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents. I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the complexity of code.
- llama-for3ver 20d agoare you able to share what you're working on that's 250k LoC?
- rolisz 20d agoIt's a life tracker, that has come to include everything but the kitchen sync (journal, workouts, food, Garmin stats, music, podcasts, books. Etc.)
- NimadFlow 20d agoThe most surprising indicator of messiness I encountered wasn't in the code itself, but in the tests. I had two tests that looked like tests but didn't actually measure anything—they simply returned True — and neither static analysis nor coverage tools caught them. The only thing that detected them was mutation testing—deliberately removing a line of code to see if the test fails; without that step, the test is essentially toothless. Also, I found that scores became inflated if you counted cases where a test passed despite the removal of code (because another test caught the issue) as a successful "detection."
- vk-candpython 20d ago[flagged]