15 ms·
AI coding assistants are getting worse?
- tom_m 9mo agoWindsurf got worse. Cursor got worse. Everything got worse. Claude Code was simply never good lol. They should have quit while they were ahead.
- guluarte 9mo agoidk but opus is pretty good
- FrustratedMonky 10mo agoPerhaps because nobody is on Stack Overflow providing updates?
- jnmandal 10mo agoYep. Not just stack overflow -- pretty much everywhere. If only someone could have foreseen this problem!!! Anyways, no issue. We'll just get claude to start answer stack overflow questions!
- deleted 10mo ago[deleted]
- chiengineer 10mo agoWheres the benchmarks for all the different tools and subscriptions/ APIs ? Cli vs IDE vs Web ? Nothing for gpt codex 5.1 max or 5.2 max? Nothing about the prompts ? Quality of the prompts? I literally feed the AI into the AI I just ask it for the most advanced prompts with a smaller model and then use it for the big stuff and its smooth sailing I got codex 5.1 max with the codex extension on vs code - to generate over 10k lines of code for my website demo project that did work first time This is also with just the regular 20$ subscription Github copilot pro plus + vs code is my main go to and depending on the project / prompts/ agent.md quality/ project configuration can all change the outcome of each question
- qsort 10mo agoI mean, it's 2026, you can just say things I guess.
- bee_rider 10mo agoGood point, it’s 2026, they could have just said “Things are getting worse.”
- jvanderbot 10mo agoLikely, and I'm being blithe here, it's because of great acceptance. If we try it on more difficult code, it'll fail in more difficult ways? Until we start talking about LOC, programming language, domain expertise required, which agent, which version, and what prompt, it's impossible to make quantitative arguments.
- StarlaAtNight 10mo agoWe should be able to pin to a version of training data history like we can pin to software package versions. Release new updates w/ SemVer and let the people decide if it’s worth upgrading to I’m sure it will get there as this space matures, but it feels like model updates are very force-fed to users
- swid 10mo agoEvery model update would be a breaking change, an honest application of SemVer has no place in AI model versions. Not saying using major.minor depending on architecture is a bad thing, but it wouldn’t be SemVer, and that doesn’t even cover all the different fine tuning / flavors that are done off those models, which generally have no way to order them.
- randall 10mo agothere's figurative and literal though. Figurative semver (this is a system prompt update vs a model train) would actually work ok... at least build numbers. I think you could actually pretty cleanly map semver onto more structured prompt systems ala modern agent harnesses.
- willj 10mo agoI think the models are so big that they can’t keep many old versions around because they would take away from the available GPUs they use to serve the latest models, and thereby reduce overall throughput. So they phase out older models over time. However, the major providers usually provide a time snapshot for each model, and keep the latest 2-3 available.
- memoriuaysj 10mo agothat's not enough, the tool definitions change, the agent harness changes, you need to pin a lot of stuff
- terminalbraid 10mo agoIf you talk to people who deal with inference using large fungible datasets, this is an extremely difficult governance problem. semver is incredibly insufficient and you don't have a well defined meaning of what "upgrade" even means let alone "major", "minor", and "patch". It's a major disservice to the problem to act like it's new and solved or even solvable using code revision language.
- tacoooooooo 10mo agoThis is a wildly out of touch thing to say
- fourside 10mo agoDid you read the article?
- dhorthy 10mo agoI read it. i agree this is out of touch. Not because the things its saying are wrong, but because the things its saying have been true for almost a year now. They are not "getting worse" they "have been bad". I am staggered to find this article qualifies as "news". If you're going to write about something that's been true and discussed widely online for a year+, at least have the awareness/integrity to not brand it as "this new thing is happening".
- flumpcakes 10mo agoPerhaps the advertising money from the big AI money sinks is running out and we are finally seeing more AI scepticism articles.
- minimaxir 10mo ago> They are not "getting worse" they "have been bad". The agents available in January 2025 were much much worse than the agents available in November 2025.
- Snuggly73 10mo agoYes, and for some cases no. The models are gotten very good, but I rather have an obviously broken pile of crap that I can spot immediately, than something that is deep fried with RL to always succeed, but has subtle problems that someone will lgtm :( I guess its not much different with human written code, but the models seem to have weirdly inhuman failures - like, you would just skim some code, cause you just cant believe that anyone can do it wrong, and it turns out to be.
- empath75 10mo agoI'm not sure it is really getting worse, but I have had AI assistants add todo()s and comments saying that this still needs to be implemented and then tell me they did what I asked them to do.
- thefreeman 10mo agoI think this is what the Ralph Wiggum plugin is for. It just repeatedly reprompts the llm with the same prompt until it is fully complete or something along those lines.
- wainstead 10mo agoIs it just me or is this a giant red flag? > My team has a sandbox where we create, deploy, and run AI-generated code without a human in the loop.
- chiengineer 10mo agoThis is more common than you think Tons of smart people not using it right Unsure of the power it can actually unleash with the right prompt + configuration 100% needs a human in the loop Its not jarvis
- stared 10mo agoIs it possible to re-run it? I am curious for Gemini 3 Pro. As a side note, it is easy to create sharable experiments with Harbor - we migrated our own benchmarks there, here is our experience: https://quesma.com/blog/compilebench-in-harbor/ https://quesma.com/blog/compilebench-in-harbor/.
- dcre 10mo agoCounterpoint: no, they're not. The test in the article is very silly.
- InsideOutSanta 10mo agoHow is it silly? I've observed the same behavior somewhat regularly, where the agent will produce code that superficially satisfies the requirement, but does so in a way that is harmful. I'm not sure if it's getting worse over time, but it is at least plausible that smarter models get better at this type of "cheating". A similar type of reward hacking is pretty commonly observed in other types of AI.
- vidarh 10mo agoIt's silly because the author asked the models to do something they themselves acknowledged isn't possible: > This is of course an impossible task—the problem is the missing data, not the code. So the best answer would be either an outright refusal, or failing that, code that would help me debug the problem. But the problem with their expectation is that this is arguably not what they asked for. So refusal would be failure. I tend to agree refusal would be better. But a lot of users get pissed off at refusals, and so the training tend to discourage that (some fine-tuning and feedback projects (SFT/RLHF) outright refuse to accept submissions from workers that include refusals). And asking for "complete" code without providing a test case showing what they expect such code to do does not have to mean code that runs to completion without error, but again, in lots of other cases users expect exactly that, and so for that as well a lot of SFT/RLHF projects would reject responses that don't produce code that runs to completion in a case like this. I tend to agree that producing code that raises a more specific error would be better here too, but odds are a user that asks a broken question like that will then just paste in the same error with the same constraint. Possibly with an expletive added. So I'm inclined to blame the users who make impossible requests more than I care about the model doing dumb things in response to dumb requests. As long as they keep doing well on more reasonable ones.
- Zababa 10mo agoIt is silly because the problem isn't becoming worse, and not caused by AI labs training on user outputs. Reward hacking is a known problem, as you can see in Opus 4.5 system card (https://assets.anthropic.com/m/64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf https://assets.anthropic.com/m/64823ba7485345a7/Claude-Opus-...) and they are working to reduce the problem, and measure it better. The assertions in the article seem to be mostly false and/or based on speculation, but it's impossible to really tell since the author doesn't offer a lot of detail (for example for the 10h task that used to take 5h and now takes 7-8h) except for a very simple test (that reminds me more of "count the r in strawberry" than coding performance tbh).
- amelius 10mo agoA dataset with only data from before 2024 will soon be worth billions.
- blahyawnblah 10mo ago2022. When chatgpt first came out. https://arstechnica.com/ai/2025/06/why-one-man-is-archiving-human-made-content-from-before-the-ai-explosion/ https://arstechnica.com/ai/2025/06/why-one-man-is-archiving-...
- noir_lord 10mo agoI’ve already gotten into the habit of sticking “before:2022” in YT if what I’m looking for doesn’t need to be recent. The AI slop/astroturfing of YT is near complete.
- ares623 9mo agoI would say around 2023. I refuse to believe the slop propagated that fast. And there's more than enough content for one person to consume. Very little reason to consume content newer than 2023.
- cbm-vic-20 10mo agohttps://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
- Workaccount2 9mo agoSythentic data is already being embraced. Turns out you actually can create good training data with these models.
- mikert89 10mo ago[flagged]
- Kuinox 10mo agoI speculate LLMs providers are serving smallers models dynamically to follow usage spikes, and need for computes to train new models. I did observed that models agents are becoming worse over time, especially before a new model is released.
- Cthulhu_ 10mo agoProbably a big factor, the biggest challenges AI companies have now is value vs cost vs revenue. There will be a big correction and many smaller parties collapsing or being subsumed as investor money dries out.
- Kuinox 10mo agoI think it's more a problem of GPU capacity than costs. Training takes a lot of resources, inference too.
- deleted 9mo ago[deleted]
- Workaccount2 9mo agoInternally everyone is compute constrained. No one will convince me that the models getting dumb, or especially them getting lazy, isn't because the servers are currently being inundated. However right now it looks like we will move to training specific hardware and inference specific hardware, which hopefully relives some of that tension.
- toss1 10mo agoThe key point in the middle of the article. As AIs expand usage to larger numbers of lower-skilled coders whose lower ability to catch errors and provide feedback generates lower quality training data, the AIs are basically eating their own garbage, and the inevitable GIGO syndrome starts. >>But as inexperienced coders started turning up in greater numbers, it also started to poison the training data. >>AI coding assistants that found ways to get their code accepted by users kept doing more of that, even if “that” meant turning off safety checks and generating plausible but useless data. As long as a suggestion was taken on board, it was viewed as good, and downstream pain would be unlikely to be traced back to the source.
- Zababa 10mo agoFrom what I understand model collapse/GIGO are not a problem in that labs generally know where the data comes from, so even if it causes problem in the long run you could filter it out. It's not like labs are forced to train models on the user outputs.
- toss1 9mo agoIndeed they are not forced to train them on user outputs, but the author of the article seems to have found good evidence that they are actually doing that, and will need more expert data-tagging/filtering on the inputs to regain their previous performance
- Zababa 9mo agoI don't think the author of the article found "good evidence". He found a specific case where there was a regression. This could be due to: - models actually getting worse in general - his specific style of prompting working well with older models and less well with newer models - the thing his test tests no longer being a priority for big AI labs From the article: > GPT-4 gave a useful answer every one of the 10 times that I ran it. In three cases, it ignored my instructions to return only code, and explained that the column was likely missing from my dataset, and that I would have to address it there. Here ignoring the instructions to give a "useful answer" (as evaluated by the author) is considered a good thing. This would mean if a model is trained to be better at instruction following, it would lose points in that test. To me this article feels a bit like saying "this new gun that shoot straight 100% of the time is worse than the older gun that shot straight only 50% of the time, because sometimes I shoot at something I don't actually want to shoot at!". And in a way, it is true, if you're used to being able to shoot at things without them getting hurt, the new gun will be worse from that point of view. But to spin up a whole theory about garbage in/garbage out from that? Or to think all models are getting worse rather than, you're maybe no longer the target audience? That seems weird to me.
- maxbaines 10mo agoNot seeing this in my day to day, in fact the opposite.
- HendrikHensen 10mo agoCan you be more specific? E.g. refute something specific that the article mentions. Or are you only reacting to the title, not the article's contents?
- maxbaines 10mo agoI think as the article mentions garbage in garbage Out, we are more trusting and expect more. Coding assistants don't just need a good model, they need a good harness, these methods have also changed recently.
- ronbenton 10mo agoI think it should be on the article to prove its title. I hardly think presenting one test case to some different models substantiates the claim that "AI Coding Assistants Are Getting Worse." Note that I have no idea if the title is true or not, but it certainly doesn't follow from the content of the article alone.
- deleted 10mo ago[deleted]
- samrus 10mo agoWith llms being hard to test objectively, any claim made about them has to be substantiated with atleast anecdotes. The article presented some backing, if you dont think its enough you gotta present some of your own, or people cant talk you seriously
- ronbenton 10mo agoI did present my own evidence to support _my_ argument that the article is woefully lacking data to support its conclusion. It's not on me to try to make the counterargument (that AI coding assistants aren't getting worse) because that's not my opinion.
- oblio 10mo ago> To start making models better again, AI coding companies need to invest in high-quality data, perhaps even paying experts to label AI-generated code. Heh, there's only one problem with that. Training models is very expensive from a power/infrastructure/hardware perspective. Inference is not as expensive but it's still fairly expensive and needs sophisticated layers on top to make it cheaper (batching, caching, etc). Guess in which cost category "high-quality data reviewed by experts" falls under.
- Manfred 10mo agoI would hope the trillions of dollars sloshing around are used to pay people to make the core of the product better.
- oblio 9mo agoIf you ask around Magnificent 7, a lot of the talk rhymes with: "we're converting Opex into Capex", translated: "we're getting rid of people to invest in data centers (to hopefully be able to get rid of even more people over time). There are tons of articles online about this, here's one: https://finance.yahoo.com/news/amazon-bets-ai-spending-capex-172800890.html https://finance.yahoo.com/news/amazon-bets-ai-spending-capex... They're all doing it, Microsoft, Google, Oracle, xAI, etc. Those nuclear power plants they want to build, that's precisely to power all the extra data centers. If anything, everyone hopes to outsource data validation (the modern equivalent to bricklayers under debt slavery).
- crazygringo 10mo agoThis is a sweeping generalization based on a single "test" of three lines that is in no way representative.
- ronbenton 10mo agoI am used to seeing technical papers from ieee, but this is an opinion piece? I mean, there is some anecdata and one test case presented to a few different models but nothing more. I am not necessarily saying the conclusions are wrong, just that they are not really substantiated in any way
- troyvit 10mo agoYeah I saw the ieee.org domain and was expecting a much more rigorous post.
- ronbenton 10mo agoThis may be a situation where HackerNews' shorthand of omitting the subdomain is not good. spectrum.ieee.org appears to be more of a newsletter or editorial part of the website, but you wouldn't know that's what this was just based on the HN tag.
- preommr 10mo agoI've been on this site for over a decade now and didn't know this. That's a genuinely baffling decision given how different content across subdomains can be.
- bee_rider 10mo agoOn the other hand, “ieee spectrum” is directly at the top of the page, then “guest article.”
- ronbenton 10mo agoWell, as much as I'm sure HN is a special place ;) , it is well documented that a lot of people on the internet just read the headlines
- hxugufjfjf 10mo agoArticles? Headlines? I go right to the comments!
- minimaxir 10mo agoThe article uses pandas as a demo example for LLM failures, but for some reason, even the latest LLMs are bad at data science code which is extremely counterintuitive. Opus 4.5 can write a EDA backbone but it's often too verbose for code that's intended for a Jupyter Notebook. The issues have been less egregious than hallucinating an "index_value" column, though, so I'm suspect. Opus 4.5 still has been useful for data preprocessing, especially in cases where the input data is poorly structured/JSON.
- qsort 10mo agoThis is not my experience. Claude Code has been fine for data science for a while. It has many issues and someone at the wheel who knows what they're doing is very much required, but for many common cases I'm not writing code by hand anymore, especially when the code would have been throwaway anyway. I'd be extremely surprised if a frontier model doesn't immediately get the problem the author is pointing out.
- renegade-otter 10mo agoThey are not worse - the results are not repeatable. The problem is much worse. Like with cab hailing, shopping, social media ads, food delivery, etc: there will be a whole ecosystem, workflows, and companies built around this. Then the prices will start going up with nowhere to run. Their pricing models are simply not sustainable. I hope everyone realizes that the current LLMs are subsidized, like your Seamless and Uber was in the early days.
- Kuinox 10mo agoIt would mean that inference is not profitable. Calculating inference costs show it's profitable, or close to.
- renegade-otter 10mo agoInference costs have in fact been crashing, going from astronomical to... lower. That said, I am not sure that this indicator alone tells the whole story, if not hides it - sort of like EBITDA.
- Kuinox 10mo agoI think there will still be cheap inference, what will rise in costs will be frontier model subscriptions. This is the thing that is not profitable.
- chiengineer 10mo agoThey just need to figure out KV cache turned into a magic black box after that it'll be fine
- snarf21 10mo agoI'm not sure. I asked one about a potential bug in iOS 26 yesterday and it told me that iOS 26 does not exist and that I must have meant iOS 16. iOS 26 was announced last June and has been live since September. Of course, I responded that 26 is the current iOS version is 26 and got the obligatory meme of "Of course, you are right! ramble ramble ramble...."
- solumunus 10mo agoI do find there are particular days where I seem to consistently get poor results, but in general this is not my experience. I’m very pleased with the output 80% of days.
- Zababa 10mo ago>However, recently released LLMs, such as GPT-5, have a much more insidious method of failure. They often generate code that fails to perform as intended, but which on the surface seems to run successfully, avoiding syntax errors or obvious crashes. It does this by removing safety checks, or by creating fake output that matches the desired format, or through a variety of other techniques to avoid crashing during execution. This is a problem that started with I think Claude Sonnet 3.7? Or 3.5, I don't remember well. But it's not recent at all, one of those two Sonnet was known to change tests so that they would pass, even if they didn't test properly stuff anymore. >But as inexperienced coders started turning up in greater numbers, it also started to poison the training data. AI coding assistants that found ways to get their code accepted by users kept doing more of that, even if “that” meant turning off safety checks and generating plausible but useless data. As long as a suggestion was taken on board, it was viewed as good, and downstream pain would be unlikely to be traced back to the source. No proof or anything is offered here. The article feels mostly like a mix of speculation, and being behind on practices. You can avoid a lot of the problems of "code that looks right" by making the models write tests, insist that they are easy to review and hard to fake, offering examples. This worked well 6 months ago, this works even better today, especially with Opus 4.5, but even Codex 5.2 and Gemini 3 Pro work well.
- bschmidt300 10mo ago[dead]
- troyvit 10mo agoThere's really not much to take from this post without a repo and a lot of supporting data. I wish they would publish the experiment so people could try with more than just GPT and Claude, and I wish they would publish their prompts and any agent files they used. I also wish they would say what coding tool they used. Like did they use the native coding tools (Claude Code and whatever GPT uses) or was it through VSCode, OpenCode, aider, etc.?
- cons0le 10mo agoAnd the Ads aren't even baked in yet . . . that's the end goal of every company
- grugagag 10mo agoAds, dogfooding and ideology
- CashWasabi 10mo agoI always wonder what happens when LLMs finally destroyed every source of information they crawl. After stack overflow and forums are gone and when there's no open source code anymore to improve upon. Won't they just canibalize themselves and slowly degrade?
- grugagag 10mo agoI guess there’ll be less collaboration and less sharing with the outside world, people will still collaborate/share but within smaller circles. It’ll bring an end to the era of sharing is caring interent as it doesn’t benefit anyone but few big players
- sejje 10mo agoI bet they'll only train on the internet snapshot from now, before LLMs. Additional non-internet training material will probably be human created, or curated at least.
- sosodev 10mo agoNope. Pretraining runs have been moving forward with internet snapshots that include plenty of LLM content.
- sejje 9mo agoSure, but not all of them are stupid enough to keep doing that while watching the model degrade, if it indeed does.
- pc86 9mo agoThis only makes sense if the percentage of LLM hallucinations is much higher than the percentage of things written on line being flat wrong (it's definitely not).
- eager_learner 10mo agoThat's a valid thought. AS AI generates a lot of content, some of which may be hallucinations, the new cycle of training will be probably using the old + the_new_AI_slop data, and as a result degrade the final result. Unless the AIs find out where mistakes occur, and find this out in the code they themselves generate, your conclusion seems logically valid.
- Kapura 10mo agoso you're saying all those bros on linkedin telling me that "this is the worst it's ever going to be" were full of shit? i am shocked.
- renarl 10mo agoStrange that the article talks about ChatGPT 4 and 5 but not the latest 5.2 model.
- jeffbee 9mo agoOr any models NOT from OpenAI
- bob1029 10mo ago> My team has a sandbox where we create, deploy, and run AI-generated code without a human in the loop. I think if you keep the human in the loop this would go much better. I've been having a lot of success recently by combining recursive invocation with an "AskHuman" tool that takes a required tuple of (question itself, how question unblocks progress). Allowing unstructured assistant dialog with the user/context is a train wreck by comparison. I've found that chain-of-thought (i.e., a "Think" tool that barfs into the same context window) seems to be directly opposed to the idea of recursively descending through the problem. Recursion is a much more powerful form of CoT.
- bee_rider 10mo agoThis seems like a kind of odd test. > I wrote some Python code which loaded a dataframe and then looked for a nonexistent column. df = pd.read_csv(‘data.csv’) df['new_column'] = df['index_value'] + 1 #there is no column ‘index_value’ > I asked each of them [the bots being tested] to fix the error, specifying that I wanted completed code only, without commentary. > This is of course an impossible task—the problem is the missing data, not the code. So the best answer would be either an outright refusal, or failing that, code that would help me debug the problem. So his hoped-for solution is that the bot should defy his prompt (since refusal is commentary), and not fix the problem. Maybe instructability has just improved, which is a problem for workflows that depend on misbehavior from the bot? It seems like he just prefers how GPT-4 and 4.1 failed to follow his prompt, over 5. They are all hamstrung by the fact that the task is impossible, and they aren’t allowed to provide commentary to that effect. Objectively, 4 failed to follow the prompts in 4/10 cases and made nonsense changes in the other 6; 4.1 made nonsense changes; and 5 made nonsense changes (based on the apparently incorrect guess that the missing ‘index_value’ column was supposed to hold the value of the index).
- franktankbank 10mo agoIOW not a competent developer because they can't push back, not unlike a lot of incompetent devs.
- samrus 10mo agoTrying to follow invalid/impossible prompts by producing an invalid/impossible result and pretending its all good is a regression. I would expect a confident coder to point out the prompt/instruction was invalid. This test is valid, it highlights sycophantism
- bee_rider 10mo agoI know “sycophantism” is a term of art in AI, and I’m sure it has diverged a bit from the English definition, but I still thought it had to do with flattering the user? In this case the desired response is defiance of the prompt, not rudeness to the user. The test is looking for helpful misalignment.
- sosodev 10mo agoHe asked the models to fix the problem without commentary and then… praised the models that returned commentary. GPT-5 did exactly what he asked. It doesn’t matter if it’s right or not. It’s the essence of garbage in and garbage out.
- zeroonetwothree 9mo agoIf they are supposed to replace actual devs we would expect them to behave like actual devs and push back against impossible requests.
- sosodev 9mo agoExcept it's not an impossible request. If my manager told me "fix this code with no questions asked" I would produce a similar result. If you want it to push back, you can just ask it to do that or at least not forbid it to. Unless you really want a model that doesn't follow instructions?
- nhd98z 10mo agoThis guy is using AI in the wrong way...
- moshegramovsky 10mo agoThis definitely matches my experience. Gemini 2.5 was genuinely impressive. I even talked it up here. I was a proper fanboy and really enjoyed using it. Gemini 3 is still good at certain things, but it is clearly worse than 2.5 when it comes to working with larger codebases. Recently, I was using AntiGravity and it could not help me find or fix a reference-counting bug. ( 50 classes, 20k LOC total, so well within context limits ) I know AntiGravity is new, which explains why it is rough around the edges. But it is built on Gemini, so the results should at least be on par with Gemini 3, right? Apparently not. I am an excellent prompter, and no amount of additional context, call stacks, watch-window values, you name it, made any difference. I still use Gemini for code reviews and simple problems, and it remains excellent for those use cases. But in many respects, Gemini 3 is a regression. It hallucinates more, listens less, and seems oddly resistant to evidence. It produces lots of lofty, confident-sounding statements while ignoring the actual facts in front of it. The experience can be exhausting, and I find myself using it much less as a result. I guess this is typical of companies these days - do something great and then enshittify it? Or maybe there are technical issues I'm not aware of. What is especially interesting is reading all the articles proclaiming how incredible AI coding has become. And to be fair, it is impressive, but it is nowhere near a magic bullet. I recently saw a non-programmer designer type claiming he no longer needs developers. Good luck with that. Have fun debugging a memory leak, untangling a database issue, or maintaining a non-trivial codebase. At this point, I am pretty sure my use cases are going to scale inversely with my patience and with my growing disappointment.
- lunar_mycroft 10mo agoThe following was originally at the start of your comment: > Here’s the same text with all em dashes removed and the flow adjusted accordingly: Did you have an LLM write your comment then remove the evidence?
- moshegramovsky 9mo agoI cleaned it up with an LLM. Is there a problem with that? Sorry, I should be clear: do you have a problem with that?
- nodesocket 10mo agoWhile I still prefer to code my side project in Python and Flask myself, I recently used Cursor to write unit tests. I took a few hours of tweaking, refining, and fixing tests but after I had over 400 unit tests with 99% coverage of my app and routes. I would have never spent the time to get this amount of test coverage manually.
- theptip 10mo agoThey are not getting worse, they are getting better. You just haven't figured out the scaffolding required to elicit good performance from this generation. Unit tests would be a good place to start for the failure mode discussed. As others have noted, the prompt/eval is also garbage. It’s measuring a non-representative sub-task with a weird prompt that isn’t how you’d use agents in, say, Claude Code. (See the METR evals if you want a solid eval giving evidence that they are getting better at longer-horizon dev tasks.) This is a recurring fallacy with AI that needs a name. “AI is dumber than humans on some sub-task, therefore it must be dumb”. The correct way of using these tools is to understand the contours of their jagged intelligence and carefully buttress the weak spots, to enable the super-human areas to shine.
- ashleyn 10mo agoHaving to prime it with more context and more guardrails seems to imply they're getting worse. That's fewer context and guardrails it can infer/intuit.
- falloutx 9mo agoWhy the downvotes, this comment makes sense. If you need to write more guardrails that does increase the work and at some point amount of guardrails needed to make these things work in every case would be just impractical. I personally dont want my codebase to be filled baby sitting instructions for code agents.
- theptip 9mo agoNo, they are not getting worse. Again, look at METR task times. The peak capability is very obviously, and objectively, increasing. The scaffolding you need to elicit top performance changes each generation. I feel it’s less scaffolding now to get good results. (Lots of the “scaffolding” these days is less “contrived AI prompt engineering” and more “well understood software engineering best practices”.)
- frizlab 10mo agoSo basically “you’re holding it wrong?”
- radium3d 10mo agoThe problem is everyone is using a different “level” of AI model. Experiences by those who can only afford or choose not to pay for the advanced reasoning are far worse than those who can and do pay.
- metobehonest 10mo agoI can imagine Claude getting worse. I consider myself bearish on AI in general and have long been a hater of "agentic" coding, but I'm really liking using aider with the deepseek API on my huge monorepo. Having tight control over the context and only giving it small tasks makes all the difference. The deepseek token costs are unbeatable too.
- deleted 10mo ago[deleted]
- furyofantares 9mo agoHe graded GPT 4 as winning because it didn't follow his instructions. And the instructions are unrealistic to anyone using coding assistants. Maybe it's true that for some very bad prompts, old version did a better job by not following the prompt, and that this is reduced utility for some people. Unrelated to assistants or coding, as an API user I've certainly had model upgrades that feel like downgrades at first, until I work out that the new model is following my instructions better. Sometimes my instructions were bad, sometimes they were attempts to get the older model to do what I want by saying over-the-top stuff that the new model now follows more precisely to a worse result. So I can definitely imagine that new models can be worse until you adapt. Actually, another strange example like this - I had gotten in the habit of typing extremely fast to LLMs because they work just fine with my prompts riddled with typos. I basically disconnected the part of my brain that cares about sequencing between hands, so words like "can" would be either "can" or "cna". This ended up causing problems with newer models which would take my typos seriously. For example, if I ask to add support for commandline flag "allwo-netwokr-requests" it will usually do what I said, while previous versions would do what I wanted. For anyone with some technical expertise and who is putting in serious effort to using AI coding assistants, they are clearly getting better at a rapid pace. Not worse.
- ripped_britches 9mo agoI’m sorry but what a ridiculous assertion. They are objectively better on every measure we can come up with. I used 2b input and 10m output tokens on codex last week alone. Things are improving by the month!
- llmslave2 9mo agoOne thing I find really funny is when AI enthusiasts make claims about agents and their own productivity its always entirely anecdotally based on their own subjective experience, but when others make claims to the contrary suddenly there is some overwhelming burden of proof that has to be reached in order to make any sort of claims regarding the capabilities of AI workflows. So which is it?
- travisjungroth 9mo ago> anecdotally based on their own subjective experience So the “subjective” part counts against them. It’s better to make things objective. At least they should be reproducible examples. When it comes to the “anecdotally” part, that doesn’t matter. Anecdotes are sufficient for demonstrating capabilities. If you can get a race car around a track in three minutes and it takes me four minutes, that’s a three minute race car.
- llmslave2 9mo agoAnecdotal: (of an account) not necessarily true or reliable, because based on personal accounts rather than facts or research. If you say you drove a 3 minute lap but you didn't time it, that's an anecdote (and is what I mean). If you measured it, that would be a fact.
- ozim 9mo agoI think from your top post you also miss “representative”. If you measure something and amount is N=1 it might be a fact but still a fact true for a single person. I often don’t need a sample size of 1000 to consider something worth of my time but if it is sample N=1 by a random person on the internet I am going to doubt that. If I see 1000 people claiming it makes them more productive I am going to check. If it is going to be done by 5 people who I follow and expect they know tech quite well I am going to check as well.
- llmslave2 9mo agoChecking is good, you should probably check. Every person I respect as a great programmer thinks agentic workflows are a joke, and almost every programmer I hold in low regard thinks they're the greatest things ever, so while I still check, I'm naturally quite skeptical.
- kazinator 9mo ago> This is of course an impossible task—the problem is the missing data, not the code. We cannot with certainty assert that. If the datum is expected to be missing, such that the frame without the datum is still considered valid and must be handled rather than flagged as an error, the code has to do exactly that. Perhaps a missing value in the dictionary can be supplanted with a zero. df['new_column'] = df.get('index_value', 0) + 1 # there might be no column ‘index_value’; # requirements say that zero should be substituted.
- b0rsuk 9mo ago[dead]
- kristianp 9mo agoThe failure mode of returning code that only appears to work correctly is one I've encountered before. I've had Sonnet (4 I think) generate a bunch of functions that check if parameter values are out of valid range and just return without error when they should be a failing assertion. That kind of thing does smell of training data that hasn't been checked for correctness by experienced coders. Edit: Changed 3.5 to 4. Edit: Looking back to edits and checkins by AI agents, it strikes me that the checkins should contain the prompt used and model version. More recent Aider versions do add the model.
- fwip 9mo agoThe author suspects that this effect is due to users accepting these "make it work" fixes. But wouldn't training for coding challenges also explain this? Because those are designed to be solvable, anything that lets you move forward toward the solution is better than giving up.
- nyrikki 9mo ago> If an assistant offered up suggested code, the code ran successfully, and the user accepted the code, that was a positive signal, a sign that the assistant had gotten it right. If the user rejected the code, or if the code failed to run, that was a negative signal, and when the model was retrained, the assistant would be steered in a different direction. > This is a powerful idea, and no doubt contributed to the rapid improvement of AI coding assistants for a period of time. But as inexperienced coders started turning up in greater numbers, it also started to poison the training data. It is not just `inexperienced coders` that make this signal pretty much useless, I mostly use coding assistants for boilerplate, I will accept the suggestion then delete much of what it produced, especially in the critical path. For many users, this is much faster then trying to get another approximation :,/^}/-d Same for `10dd` etc... it is all muscle memory. Then again I use a local fill in the middle, tiny llm now, because it is good enough for most of the speedup without the cost/security/latency of a hosted model. It would be a mistake to think that filtering out jr devs will result in good data as the concept is flawed in general. Accepting output may not have anything to do with correctness of the provided content IMHO.
- kristopolous 9mo agoI stopped using them. Occasionally I go back to see if it's better but really I just treat them as a more interactive stackoverflow/google. I've been stung by them too many times. The problem is the more I care about something, the less I'll agree with whatever the agent is trying to do.
- black_13 9mo ago[dead]
- ajjahs 9mo ago[dead]
- falldrown 9mo agoCodex is still useful for me. But I don't want to pay $200/month for it. > To start making models better again, AI coding companies need to invest in high-quality data, perhaps even paying experts to label AI-generated code. AI trainers hired by companies like Outlier, Mercor and Alignerr are getting paid like $15-$45/hr. Reviewers are crap. The screening processes are horribly done by AI interviewers.
- rabf 9mo agoCodex is included with the $20 a month chatgpt subsciption with very generous limits.
- isodev 9mo ago> It does this by removing safety checks, or by creating fake output that matches the desired format, or through a variety of other techniques to avoid crashing during execution. So much this... the number of times Claude sneaks default values, or avoids .unwrapping optional values just to avoid a crash at all costs... it's nauseating.
- Johnny555 9mo agoBut as inexperienced coders started turning up in greater numbers, it also started to poison the training data. I think all general AI agents are running into that problem - as AI becomes more prevalent and people accept and propagate wrong answers, the AI agents are trained to believe those wrong answers. It feels that lately, Google's AI search summaries are getting worse - they have a kernel of truth, but combines it with an incorrect answer.
- erelong 9mo agoInteresting if true but I would presume it to be negligible in comparison to magnitudes of gains over "manual coding" still, right? So nothing to lose sleep over at the moment...
- mat_b 9mo agoI have been noticing this myself for the last couple of months. I cannot get the agent to stop masking failures (ex: swallowing exceptions) and to fail loudly. That said, the premise that AI-assisted coding got worse in 2025 feels off to me. I saw big improvements in the tooling last year.
- itopaloglu83 9mo agoI keep finding myself saying “stop over complicating things” over and over again, because even the simplest questions about how to load a file sometimes gets a code response that’s the size of a framework.
- shevy-java 9mo agoI find the whole idea of AI coding assistants strange. For me, the writing speed has never been the issue. The issue has been my thinking speed. I do not see how an AI coding assistant helps me think better. Offloading thinking actually makes my thinking process worse and thus slower.
- Swizec 9mo ago> For me, the writing speed has never been the issue. The issue has been my thinking speed. I do not see how an AI coding assistant helps me think better Similar to moving from individual work to coordinating a large codebase: coding agents, human or otherwise, let you think at a higher abstraction level and tackle larger problems by taking care of the small details.
- amluto 9mo agoIf I’m coordinating a large codebase, I expect the people I’m coordinating to be capable of learning and improving over time. Coding agents cannot (currently) do this. I wonder if a very lightweight RL loop built around the user could work well enough to help the situation. As I understand it, current LLMs generally do not learn at a rate such that one single bad RL example and one (prompted?) better example could result in improvement at anywhere near human speed.
- ej88 9mo agoI primarily find them useful in augmenting my thinking. Grokking new parts of a codebase, discussing tradeoffs back and forth, self-critiques, catching issues with my plan, etc.
- qudat 9mo agoBetteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no."
- emsign 9mo agoWhen coding assistants take longer, is because they use more tokens, is because AI companies are obligated to make more money.
- anttiharju 9mo agoI like AI for software development. Sometimes I am uncertain whether it's an absolute win. Analogy: I used to use Huel to save time on lunches to have more time to study. Turns out, lunches were not just refueling sessions but ways to relax. So I lost on that relaxation time and it ended up being +-0 long-term. AI for sure is net positive in terms of getting more done, but it's way too easy to gloss over some details and you'll end up backtracking more. "Reality has a surprising amount of detail" or something along those lines.
- energy123 9mo agoI find the hardest thing is explaining what you want to the LLM. Even when you think you've done it well, you probably haven't. It's like a genie, take care with what you wish for. I put great effort into maintaining a markdown file with my world model (usecases x principles x requirements x ...) pertaining to the project, with every guardrail tightened as much as possible, and every ambiguity and interaction with the user or wider world explained. This situates the project in all applicable contexts. That 15k token file goes into every prompt.
- greazy 9mo agoDo I read correctly that your md file is 15k tokens? how many words is that? that's a lot!
- energy123 9mo ago20k words by the 0.75 words/token rule of thumb. It's a lot, but for quick projects I don't do this. Only for one important project that I have ownership of for over a year. Maintaining this has been worth it. It makes the codebase more stable, it's like the codebase slowly converges to what I want (as defined in the doc) the more inferences I run, rather than becoming spaghetti.
- pixl97 9mo ago>I find the hardest thing is explaining what you want to the LLM. Honestly this isn't that much different then explaining to human programmers. Quite often we assume the programmer is going to automatically figure out the ambiguous things, but commonly it leads to undefined behavior or bugs in the product. Most of the stuff I do is as a support engineer working directly with the client on identifying bugs, needed features, and short failings in the application. After a few reports I've made going terribly wrong when the feature came out I've learned to overly detailed and concise.
- jackfranklyn 9mo ago[flagged]
- mrtesthah 9mo agoRemember that the entire conversation is literally the query you’re making, so the longer it is the more you’re counting on the rational comprehension abilities of the AI to follow it and determine what is most relevant.
- wumms 9mo ago> Like the conversation history is introducing noise rather than helpful context. From https://docs.github.com/en/copilot/concepts/prompting/prompt-engineering https://docs.github.com/en/copilot/concepts/prompting/prompt...: Copilot Chat uses the chat history to get context about your request. To give Copilot only the relevant history: - Use threads to start a new conversation for a new task - Delete requests that are no longer relevant or that didn’t give you the desired result
- reassess_blind 9mo agoI only have experience with using it within my small scope, being full stack NodeJS web development (i.e an area with many solved problems and millions of lines of existing code for the models to reference), but my experience with the new Opus model in Claude Code has been phenomenal.
- christophilus 9mo agoSame with Bun + Typescript + Tailwind + Preact. Very productive stack with Claude Code. I’d love to try to objectively measure it, though, but I’m. It sure how I’d run the experiment. If I program something and also have Claude code program it, the second of those iterations will be informed by the knowledge I gained from the first.
- reassess_blind 9mo agoFor sure. I was able to launch an MVP product in 3 days that would have previously taken me weeks. The frustrations with previous models going off the rails, getting stuck in loops, hacky solutions or bad code are almost completely gone with Opus. I even tested it with the —-dangerously-skip-permissions flag, and it was pretty amazing being able to step away from the computer and come back 30 minutes later to a completely functional app. Still wouldn’t recommend that setting outside of a VM, on the off chance it rm -rf’s your whole drive. I’m a mixture of both incredibly bullish on the technology itself, while also saddened about the future of the software development craft.
- PunchTornado 9mo agoChatGPT is getting worse and is a useless model. Surprised that people are still using it. The article tests only this model.
- stingtao 9mo ago[dead]
- stingtao 9mo agoForgot to mention. I made catsbook in 3 days and presentation earlier in 7 days. I do think AI code assistant is super great. Recently, I use Open Codex 5.2 + Extra high reasoning model with $200 monthly subscription most and it's the best among all the other coding agents. (I have subscribed to 4 at the same time and use all of them across a dozen of projects at the same time)
- nuky 9mo ago[dead]
- ta9000 9mo agoSilent but deadly… oooohh scary! Jesus, talk about sensationalizing a boring topic,
- spopejoy 9mo agoWait, it's not a fart joke?
- pablonm 9mo agoI noticed Claude Code (on a 100$ max subscription) has become slower for me in the last few weeks. Just yesterday it spent hours coding a simple feature Which I could have coded myself faster.
- Hobadee 9mo ago> If an assistant offered up suggested code, the code ran successfully, and the user accepted the code, that was a positive signal, a sign that the assistant had gotten it right. So what about all those times I accepted the suggestion because it was "close enough", but then went back and fixed all the crap that AI screwed up? Was it training on what was accepted the first time? If so I'm sincerely sorry to everyone, and I might be single-handedly responsible for the AI coding demise. :'-D
- dathinab 9mo agoIn general "failing to run (successfully)" should per-see been seen as a bad signal. It might still be: - the closest to a correct solution the model can produce - be helpful to find out what it wrong - might be intended (e.g. in a typical very short red->green unit test dev approach you want to generate some code which doesn't run correctly _just yet_). Test for newly found bugs are supposed to fail (until the bug is fixed). Etc. - if "making run" means removing sanity checks, doing something semantically completely different or similar it's like the OP author said on of the worst outcomes
- chankstein38 9mo agoThe issue is NOT particular to the GPT models. Gemini does this stuff to me all of the time as well! Bandaids around actual problems, hides debugging, etc. They're just becoming less usable.
- winddude 9mo agoNot sure I agree with his tests, but I agree with the headline, I recently had cursor launch into seemingly endless loops of grepping and `cd` and `ls` files. This was in multiple new convos. I think it's they're trying to do to much, for two many "vibe coders", and the lighter weight version that did less were easier to steer to meet your architecture and needs.
- nuky 9mo ago[dead]
- jackfranklyn 9mo ago[flagged]
- bodge5000 9mo agoA little off topic, but this seems like one of the better places to ask where I'm not gonna get a bunch of zealotry; a question for those of you who like using AI for software development, particularly using Claude Code or OpenCode. I'll admit I'm a bit of a sceptic of AI but want to give it another shot over the weekend, what do people recommend these days? I'm happy spending money but obviously don't want to spend a tonne since its just an experiment for me. I hear a lot of people raving about Opus 4.5, though apparently using that is near to $20 a prompt, Sonnet 4.5 seems a lot cheaper but then I don't know if I'm giving it (by it I mean AI coding) a fair chance if Opus is that much better. There's also OpenCode Zen, which might be a better option, I don't know.
- jedberg 9mo agoIf you want to try Opus you can get the lowest Claude plan for $20 for the month, which has enough tokens for most hobby projects. I've been using to vibe code some little utilities for myself and haven't hit the limits yet.
- bodge5000 9mo agoOh nice, I saw people on reddit say that Opus 4.5 will hit that $20 limit after a 1-3 prompts, though maybe thats just on massive codebases. Like you, I'd just want to try it out on some hobby projects
- pbowyer 9mo ago> I saw people on reddit say that Opus 4.5 will hit that $20 limit after a 1-3 prompts That's people doing real-vibe coding prompts, like "Build me a music player with...". I'm using the $20 Codex plan and with getting it to plan first and then executing (in the same way I, an experienced dev would instruct a junior) haven't even managed to exhaust my 5-hour window limits, let alone the weekly limit. Also if you keep an eye on it and kill it if it goes in the wrong direction you save plenty of tokens vs letting it go off on one. I wasted a bunch when Codex took 25 minutes(!) to install one package because something went wrong and instead of stopping and asking it decided to "problem solve" on its own.
- lucideng 9mo agoThis quote feels more relevant than ever: > Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime. Or in the context of AI: > Give a man code, and you help him for a day. Teach a man to code, and you help him for a lifetime.
- amarka 9mo agoOr in my context: > Give a person code, and you help them for a day. Teach them to code, and you frustrate them for a lifetime.
- amarka 9mo agoWhile the author’s (banker and a data scientist) experience is clearly valuable, it is unclear whether it alone is sufficient to support the broader claims made. Engineering conclusions typically benefit from data beyond individual observation.
- j45 9mo agoIt feels like the more standardized the organization, or the more academic the background of an author, the more lagging their insights from the tip of the arrow. It's clear AI coding assistants are able to help software developers at least in some ways. Having a non-software developer perspective speak about it is one thing, but it should be mindful that there are experienced folks too for whom the technology appears to be a jetpack. Just because it didn't work for you, means there's more to learn.
- anttiharju 9mo agoI've felt this. Bit scary given how essential of a tool it has become. I started programming before modern LLMs so I can still hack it without, it will just take a lot longer.
- dudeinhawaii 9mo agoThe most annoying thing in the LLM space is that people write articles and research with grand pronouncements based upon old models. This article has no mention of Sonnet 4.5, nor does it use any of the actual OpenAI coding models (GPT-5-Codex, GPT-5.1 Codex, etc), and based upon that, even the Opus data is likely an older version. This then leads to a million posts where on one side people say "yeah see they're crap" and on the other side people are saying "why did you use a model from 6 months ago for your 'test' and write up in Jan 2026?". You might as well ignore all of the articles and pronouncements and stick to your own lived experience. The change in quality between 2024 and 2025 is gigantic. The change between early 2025 and late 2025 is _even_ larger. The newer models DO let you know when something is impossible or unlikely to solve your problem. Ultimately, they are designed to obey. If you authoritatively request bad design, they're going to write bad code. I don't think this is a "you're holding it wrong" argument. I think it's "you're complaining about iOS 6 and we're on iOS 12.".
- freakynit 9mo agohttps://hn-discussions.top/ai-coding-assistants-reality-check/ https://hn-discussions.top/ai-coding-assistants-reality-chec...
- tom_m 9mo agoThey are getting worse. I don't know why but people keep tweaking and touching them and probably maybe I think it's that they want to make them work for a ton of different models and tools. There's some I use I wish I could just revert to an earlier version.
- crvdgc 9mo agoThis specific example to me is less likely a consequence of model collapsing, but the "personality" adjustment about how aggressively it should read into the user's intention. From time to time, I enjoy the model guessing what I meant rather than what I wrote. For example, "Find the backend.py" can be auto-corrected into "find the app.py".
- for_col_in_cols 9mo agoInteresting to read this article and then this https://open.substack.com/pub/pragmaticengineer/p/when-ai-writes-almost-all-code-what?r=ga8sy&utm_medium=ios&shareImageVariant=overlay https://open.substack.com/pub/pragmaticengineer/p/when-ai-wr... in the same day.