5 ms·

I think AI-assisted programming may be having the opposite effect, at least for me. I'm now incentivized to use less abstractions. Why do we code with React?

by simonw 11mo ago

I think AI-assisted programming may be having the opposite effect, at least for me. I'm now incentivized to use less abstractions. Why do we code with React? It's because synchronizing state between a UI and a data model is difficult and it's easy to make mistakes, so it's worth paying the React complexity/page-weight tax in order for a "better developer experience" that allows us to build working, reliable software with less typing of code into a text editor. If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. How often have you dropped in a big complex library like Moment.js just because you needed to convert a time from one format to another, and it would take too long to hand-write that one feature (and add tests for it to make sure it's robust)? With an LLM that's a single prompt and a couple of minutes of wait. Using LLMs to build black box abstraction layers is a choice. We can choose to have them build LESS abstraction layers for us instead.

tyre 11mo ago
For moment you an use `date-fns` and tree shake. I'd rather have LLMs build on top of proven, battle-tested production libraries than keep writing their own from scratch. You're going to fill up context with all of its re-invented wheels when it already knows how to use common options. Not to mention that testing things like this is hard. And why waste time (and context and complexity) for humans and LLMs trying to do something hard like state syncing when you can focus on something else?
- simonw 11mo ago
  Every dependency carries a cost. You are effectively outsourcing part of the future maintenance of your project to an external team. This can often be a very solid bet, but it can also occasionally backfire if the library you chose falls out of date and is no longer maintained. For this reason I lean towards fewer dependencies, and have a high bar for when a dependency is worth adding to a project. I prefer a dozen well vetted dependencies to hundreds of smaller ones that each solve a problem that I could have solved effectively without them.
  - tyre 11mo ago
    For smol things like left-pad, sure but the two examples given (moment and react) solve really hard problems. If I were reviewing a PR where someone tried to re-implement time zone handling in JS, that’s not making it through review. In JS, the DOM and time zones are some of the most messed up foundations you’re building on top of ime. (The DOM is amazing for documents but not designed for web apps.) I think we really need to be careful about adding dependencies that we’re maintaining ourselves, especially when you factor in employee churn and existing options. Unless it’s the differentiator for the business you’re building, my advice to engineers is to strongly consider other options and have a case for why they don’t fit. AI can play into the engineering blind spot of building it ourselves because it’s fun. But engineering as a discipline requires restraint.
    - simonw 11mo ago
      Whether that's true about React and Moment varies on a case-by-case basis. If you're building something simple like a contact form React may not be the right choice. If you're building something like Trello that calculation is different. Likewise, I wouldn't want Moment for https://tools.simonwillison.net/california-clock-change https://tools.simonwillison.net/california-clock-change but I might want it for something that needs its more advanced features.
  - deleted 11mo ago
    [deleted]
azangru 11mo ago
> Why do we code with React? ...is a loaded question, with a complex and nuanced answer. Especially when you continue: > it's worth paying the React complexity/page-weight tax All right; then why do we code in React when a smaller alternative, such as Preact, exists, which solves the same problem, but for a much lower page-weight tax? Why do we code in React when a mechanism to synchronize data with tiny UI fragments through signals exists, as exemplified by Solid? Why do people use React to code things where data doesn't even change, or changes so little that to sync it with the UI does not present any challenge whatsoever, such as blogs or landing pages? I don't think the question 'why do we code with React?' has a simple and satisfactory answer anymore. I am sure marketing and educational practices play a large role in it.
- simonw 11mo ago
  Yeah, I share all of those questions. My cynical answer is that most web developers who learned their craftsin the last decade learned frontend React-first, and a lot of them genuinely don't have experience working without it. Which means hiring for a React team is easier. Which means learning React makes you more employable.
  - whstl 11mo ago
    > most web developers who learned their craftsin the last decade learned frontend React-first, and a lot of them genuinely don't have experience working without it That's not cynical, that's the reality. I do a lot of interviews and mentor juniors, and I can 100% confirm that. And funny enough, React-only devs was a bigger problem 5 years ago. Today the problem is developers who can *only* use Next.js. A lot can't use Vite+React or plain React, or whatever. And about 50% of Ruby developers I interviewed from 2022-2024 were unable to code a FizzBuzz in Ruby without launching a whole Rails project.
    - CharlieDigital 11mo ago
      My test for FE is to write a floating menu in JSFiddle with only JS, CSS, and HTML. Bonus if no JS. If you can do that, then you can probably understand how everything else works.
sshine 11mo ago
> I'm now incentivized to use less abstractions. I'm incentivised to use abstractions that are harder to learn, but execute faster or more safely once compiled. E.g. more Rust, Lean. > If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. LLMs benefit from abstractions the same way as we do. LLMs currently copy our approaches to solving problems and copy all the problems those approaches bring. Letting LLMs skip all the abstractions is about as likely to succeed as genetic programming is efficient. For example, writing more vanilla JS instead of React, you're just reinventing the necessary abstractions more verbosely and with a higher risk of duplicate code or mismatching abstractions. In a recent interview with Bret Weinstein, a former professor of evolutionary biology, he proposed that one property of evolution that makes the story of one species evolving into another more likely is that it's not just random permutations of single genes; it's also permutations to counter variables encoded as telomeres and possibly microsatellites. https://podcasts.happyscribe.com/the-joe-rogan-experience/2427-bret-weinstein https://podcasts.happyscribe.com/the-joe-rogan-experience/24... Bret compares this to flipping random bits in a program to make it work better vs. tweaking variables randomly in a high-level language. Mutating parameters at a high-level for something that already works is more likely to result in something else that works than mutating parameters at a low level. So I believe LLMs benefit from high abstractions, like us. We just need good ones; and good ones for us might not be the same as good ones for LLMs.
- simonw 11mo ago
  > For example, writing more vanilla JS instead of React, you're just reinventing the necessary abstractions more verbosely and with a higher risk of duplicate code or mismatching abstractions. Right, but I'm also getting pages that load faster and don't require a build step, making them more convenient to hack on. I'm enjoying that trade-off a lot.
  - joquarky 11mo ago
    Vanilla JS is also a lot more capable than it was when React was invented. And yeah, you can't beat the iteration speed. I feel like there are dozens of us.
starkparker 11mo ago
I'd rather use React than a bespoke solution created by an ephemeral agent, and I'd rather self-trepanate than use React
roadside_picnic 11mo ago
> If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. I've had plenty of junior devs justify massive code bases of random scripts and 100+ line functions with the same logic. There's a reason senior devs almost always push back on this when it's encountered. Everything hinges on that "if". But you're baking a tautology into your reasoning: "if LLMs can do everything we need them to, we can use LLMs for everything we need". The reason we stop junior devs from going down this path is because experience teaches us that things will break and when they do, it will incur a world of pain. So "LLM as abstraction" might be a possible future, but it assumes LLMs are significantly more capable than a junior dev at managing a growing mess of complex code. This is clearly not the case with simplistic LLM usage today. "Ah! But you need agents and memory and context management, etc!" But all of these are abstractions. This is what I believe the parent comment is really pointing out. If AI could do what we originally hoped it could: follow simple instructions to solve complex tasks. We'd be great, and I would agree with your argument. But we are very clearly not in that world. Especially since Karpathy can't even keep up with the sophisticated machinery necessary to properly orchestrate these tools. All of the people decrying "you're not doing it right!" are emphatically proving that LLMs cannot perform these tasks at the level we need them to.
- mannanj 11mo ago
  > things will break and when they do, it will incur a world of pain How much if this is still true and exaggerated in our world environment today where the cost of making things is near 0? I think “Evolution” would say that the cost of producing is near 0 so the possibility of creating what we want is high. The cost of trying again is low so mistakes and pain aren’t super high. For really high stakes situation (which most situations are not) bring the expert human in the loop until the expert better than that human is AI.
- neoromantique 11mo ago
  I'm sorry, but I don't agree. Current dependency hell that is modern development, just how wide the openings are for supply chain attacks and seemingly every other week we get a new RCE. I'd rather 100 loosely coupled scripts peer reviewed by a half a dozen of LLM agents.
nzoschke 11mo ago
Right there with you. I'm instructing my agents to doing old school boring form POST, SSR templates, and vanilla JS / CSS. I previously shifted away from this to abstractions because typing all the boilerplate was tedious. But now that I'm not typing, the tedious but simple approach is great for the agent writing the code, and great for the the people doing code reviews.
jayd16 11mo ago
Why would I want to maintain in perpetuity random snippets when a library exists? How is that an improvement?
- simonw 11mo ago
  It's an improvement if that library stops being actively maintained in the future. ... or decides to redesign the API you were using.
  - skylurk 11mo ago
    Are you referring to httpx? ;)
oulipo2 11mo ago
That's a fundamental misunderstanding The role of abstractions *IS* to prevent (eg "compress") the need for a test suite, because you have an easy model to understand and reason about
- simonw 11mo ago
  One of my personal rules for automated test suites is that my tests should fail if one of the libraries I'm using changes in a way that breaks my features. Makes upgrading dependencies so much less painful!
  - oulipo2 11mo ago
    Of course, but this is largely unmaintainable, shifting the responsibility of correctness check from libraries to users. That's why we modularize/abstract/simplify, in order to minimize the need for actual checks
casualscience 11mo ago
If you work at a megacorp right now, you know whats happening isn't people deciding to use less libraries. It's developers being measured by their lines of code, and the more AI you use the more lines of code and 'features' you can ship. However, the quality of this code is fucking terrible, no one is reading what they push deeply, and these models don't have enough 'sense' to make really robust and effective test suites. Even if they did, a comprehensive test suite is not the solution to poorly designed code, it's a band aid -- and an expensive one at scale. Most likely we will see some disasters happening in the next few years due to this mode of software development, and only then will people understand to use these agents as tools and not replacements. ...Or maybe we'll get AGI and it will fix/maintain the trash going out there today.
akoboldfrying 11mo ago
> and it can maintain a test suite that shows everything works correctly Are you able to efficiently verify that the test suite is testing what it should be testing? (I would not count "manually reviewing all the test code" as efficient if you have a similar amount of test code to actual code.) Sometimes a change to the code under test means that a (perhaps unavoidably brittle) test needs to be changed. In this case, the LLM should change the test to match the behaviour of the code under test. Other times, a change to the code under test represents a bug that a failing test should catch -- in this case, the LLM should fix the code under test, and leave the test unchanged. How do you have confidence that the LLM chooses the right path in each case?
travisgriggs 11mo ago
Has anyone tried the experiment that is sort of implied here? I was wondering earlier today, what it would be like to pick a simple app, pick on OS, and just tell an LLM to write that app using only machine code and native ADKs, and skip all intermediate layers? We seem to have created a large bureaucracy for software development, where telling a computer how to execute an app involves keeping a lot of cogs in a big complicated machine happy. But why use the automation to just roll the cogs? Why not just simplify/streamline? Does an LLM need to worry about using the latest and greatest abstractions? I have to assume this has been tried already...
nkrisc 11mo ago
If LLMs are that capable, then why are AI companies selling access to them instead of using them to conquer markets?
- lithocarpus 11mo ago
  I would guess fear of losing market share and valuable data, as well as pressure to appear to be winning the AI race for the companies' own stock price. i.e competition. If there were only one AI company, they would probably not release anything close to their most capable version to the public. ala Google pre-chatgpt.
  - tjr 11mo ago
    I’m not sure that really answers the question? Or perhaps my interpretation of the question is different. If (say) the code generation technology of Anthropic is so good, why be in the business of selling access to AI systems? Why not instead conquer every other software industry overnight? Have Claude churn out the best office application suite ever. Have Claude make the best operating system ever. Have Claude make the best photo editing software, music production software, 3D rendering software, DNA analysis software, banking software, etc. Why be merely the best AI software company when you can be the best at all software everywhere for all time?
    - sod22 11mo ago
      Im waiting for people to realise that software products are much more than just lines of code. Getting sick and tired of people talk about their productivity gains when not much is actually happening out there in terms of real value creation.
      - pertymcpert 11mo ago
        Just because you don't see it or refuse to believe people doesn't make you right and them liars. Maybe you're just wrong.
rdhatt 11mo ago
I find it interesting for your example you chose Moment.js -- a time library instead of something utilitarian like Lodash. For years I've following Jon Skeet's blog about implementing his time library NodaTime (a port of JodaTime). There are a crazy number of edge cases and many unintuitive things about modeling time within a computer. If I just wanted the equivalent of Lodash's _.intersection() method, I get it. The requirements are pretty straightforward and I can verify the LLM code & tests myself. One less dependency is great. But with time, I know I don't know enough to verify the LLM's output. Similar to encryption libraries, it's a common recommendation to leave time-based code to developers who live and breathe those black boxes. I trust the community verify the correctness of those concepts, something I can't do myself with LLM output.
cyberax 11mo ago
The problem is, what do you do _when_ it fails? Not "if", but "when". Can you manually wade through thousands of functions and fix the issue?
losvedir 11mo ago
Huh, I've been assuming the opposite: better to use React even if you don't need it, because of its prevalence in the training data. Is it not the case that LLMs are better at standard stacks like that than custom JS?
- simonw 11mo ago
  Hard to say for sure. I've been finding that frontier LLMs write very good code when I tell them "vanilla JS, no React" - in that their code matches my personal taste at least - but that's hardly a robust benchmark.
api 11mo ago
Nutty idea: train on ASM code. Create an LLM that compiles prompts directly to machine code.
majormajor 11mo ago
> If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. for simple stuff, sure, React was ALWAYS inefficient. Even Javascript/client-side logic is still overkill a lot of the times except for that pesky "user expectations" thing. for anything codebase that's long-lived and complex, combinatorics tells us how it'll near-impossible to have good+fast test coverage on all that. part of the reason people don't roll their own is because being able to assume that the library won't have major bugs leads to an incredible reduction in necessary test service, and generally people have found it a safe-enough assumption. throwing that out and trying to just cover the necessary stuff instead - because you're also throwing out your ability to quickly recognize risky changes since you aren't familiar with all the code - has a high chance of painting you into messy corners. "just hire a thousand low-skilled people and force them to write tests" had more problems as a hiring plan then just "people are expensive."
godelski 11mo ago
> I'm now incentivized to use less abstractions. I'd argue it's a different category of abstraction
throwaway150 11mo ago
> If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. But this is a highly non-trivial problem. How do you even possibly manually verify that the test suite is complete and tests all possible corner cases (of which there are so many because synchronizing state is a hard problem)? At least React solves this problem in a non-stochastic, deterministic manner. What can be a good reason to replace something like React that works determinstically with LLM-assisted code that is generated stochastically and there's no easy way to manually verify if the implementation or the test suite is correct and complete?
- mlinhares 11mo ago
  You don't, same as for the "generate momentjs and use it". People now firmly believe they can use an LLM to build custom versions of these libraries and rewrite whole ecosystems out of nowhere because Claude said "here's the code". I've come to realize fighting this is useless, people will do this, its going to create large fuck ups and there will be heaps of money to be made on the cleanup jobs.
  - pertymcpert 11mo ago
    There's going to be lots of fuck ups, but with frontier models improving so much there's also going to be lots of great things made. Horrible, soul crushing technical debt addressed because it was offloaded to models rather than spending a person's thought and sanity on it. I think overall for engineering this is going to be a net positive.
  - bonesss 11mo ago
    I think the gap between people dealing with JavaScript cruft all day and backend large systems development is creating a massive conversational disconnect… like, this thread is plain-faced and seriously discussing reinventing date handling locally for funsies. I also think that any company creating a reverse-centaur workforce of blind and dumb half baked devs ritualistically shaking chicken bones at their pay-as-you-go automaton has effectively outsourced their core business to OpenAI/MS while paying for the privilege. And, on the twenty year timeline as service and capital costs create crunches, those mega corps will literally be sitting on whole copies of internal business schematics and critical code of their subservient customers… They say things, they do other things. Trusting Microsoft not to eat your sector through abusive partner programs and licensing entanglements backed with government capture? Surely the LLMs can explain how that has gone historically and how smart that is going forward.
avaika 11mo ago
I don't trust LLM enough to handle the maintenance of all the abstraction buried in react / similar library. I caught some of the LLMs taking nasty shortcuts (e.g. removing test constraints or validations in order to make the test green). Multiple times. Which completely breaks trust. And if I have to closely supervise every single change, I don't believe my development process will be any better. If not worse. Let alone new engineers who join the team and all of a sudden have to deal with a unique solution layer which doesn't exist anywhere else.
qingcharles 11mo ago
LLMs also have encyclopedic knowledge. Several times LLMs have found some huge block of code I wrote and reduced it down to a few lines. The other day they removed several thousand lines of brittle code I wrote previously for some API calls with a well-tested package I didn't know about. Literally thousands down to dozens. My code is constantly shrinking, becoming better quality, more performant, more best-practice on a daily basis. And I'm learning like crazy. I'm constantly looking up changes it recommends to see why and what the reasons are behind them. It can be a big damned dummy too, though. Just today it was proposing a massive server-side script to workaround an issue with my app I was deploying, when the actual solution was to just make a simple one-line change to the app. ("You're absolutely right!")
briandw 11mo ago
I've come to a similar conclusion. One example is how much easier it is to put an interface on top of sqlite. I've been burned badly with the hidden details of ORM s. ORMs are the sirens call of getting rid of all that boiler plate code when encoding and decoding objects into a db. However this abstraction breaks in many hidden ways. Lazy loading details, in-memory state vs db mismatch, cascading details, etc all have unexpected problems that can be hard to predict. Using an LLM to do the grunt work lets you easily see and reason about all the details. You don't have to guess about what's happening and you can make your own choices.
fweimer 11mo ago
> If an LLM is typing that code - and it can maintain a test suite that shows everything works correctly - maybe we don't need that abstraction after all. I'm worried that there is a tendency in LLM-generated code to avoid even local abstractions, such as putting common code into separate (local functions), and even use records/structures. You end up with code that is best maintained with an LLM, which is good for the LLM provider and their future revenue. But we humans as reviewers and ultimate long-term maintainers benefit from those minor abstractions.
- simonw 11mo ago
  Yeah, I find myself needing to watch out for that. I'll frequently say "refactor that to reduce duplicated code" - which is generally very safe once the LLM has added test coverage for the new feature.

simonw 11mo ago

I'm not arguing for using LLMs as an abstraction. I'm saying that a key component of the dependency calculation has changed. It used to be that one of the most influential facts affecting your decision to add a new library was the cost of writing the subset of code that you needed yourself. If writing that code and the accompanying tests represented more than an hour of work, a library was usually a better investment. If the code and tests take a few minutes those calculations can look very different. Making these decisions effectively and responsibly is one of the key characteristics of a senior engineer, which is why it's so interesting that all of those years of intuition are being disrupted. The code we are producing remains the same. The difference is that a senior developer may have written that function + tests in several hours, at a cost of thousands of dollars. Now that same senior developer can produce exactly the same code at a time cost of less than $100.

brians 11mo ago
A major difference is when we have to read and understand it because of a bug. Perhaps the LLM can help us find it! But abstraction provides a mental scaffold
- godelski 11mo ago
  I feel like "abstraction" is overloaded in many conversations. Personally I love abstraction when it means "generalize these routines to a simple and elegant version". Even if it's harder to understand than a single instance it is worth the investment and gives far better understanding of the code and what it's doing. But there's also abstraction meaning to make less understandable or more complex and I think LLMs operate this way. It takes a long time to understand code. Not because any single line of code is harder to understand but because they need to be understood in context. I think part of this is in people misunderstanding elegance. It doesn't mean aesthetically pleasing, but to do something in a simple and efficient way. Yes, write it rough the first round but we should also strive for elegance. It more seems like we are just trying to get the first rough draft and move onto the next thing.
all_factz 11mo ago
React is hundreds of thousands of lines of code (or millions - I haven’t looked in awhile). Sure, you can start by having the LLM create a simple way to sync state across components, but in a serious project you’re going to run into edge-cases that cause the complexity of your LLM-built library to keep growing. There may come a point at which the complexity grows to such a point that the LLM itself can’t maintain the library effectively. I think the same rough argument applies to MomentJS.
- simonw 11mo ago
  If the complexity grows beyond what it makes sense to do without React I'll have the LLM rewrite it all in React! I did that with an HTML generation project to switch from Python strings to Jinja templates just the other day: https://github.com/simonw/claude-code-transcripts/pull/2 https://github.com/simonw/claude-code-transcripts/pull/2
  - zdragnar 11mo ago
    That would go over on any decently sized team like a lead balloon.
    - simonw 11mo ago
      As it should, normally, because "we'll rewrite it in React later" used to represent weeks if not months of massively disruptive work. I've seen migration projects like that push on for more than a year! The new normal isn't like that. Rewrite an existing cleanly implemented Vanilla JavaScript project (with tests) in React the kind of rote task you can throw at a coding agent like Claude Code and come back the next morning and expect most (and occasionally all) of the work to be done.
      - zdragnar 11mo ago
        And everyone else's work has to be completely put on hold or thrown away because you did the whole thing all at once on your own. That's definitely not something that goes over well on anything other than an incredibly trivial project.
        pertymcpert 11mo ago
        Why did you jump to the assumption that this: > The new normal isn't like that. Rewrite an existing cleanly implemented Vanilla JavaScript project (with tests) in React the kind of rote task you can throw at a coding agent like Claude Code and come back the next morning and expect most (and occasionally all) of the work to be done. ... meant that person would do it in a clandestine fashion rather than this be an agreed upon task prior? Is this how you operate?
squigz 11mo ago
> Making these decisions effectively and responsibly is one of the key characteristics of a senior engineer, which is why it's so interesting that all of those years of intuition are being disrupted. They're not being disrupted. This is exactly why some people don't trust LLMs to re-invent wheels. It doesn't matter if it can one-shot some code and tests - what matters is that some problems require experience to know what exactly is needed to solve that problem. Libraries enable this experience and knowledge to centralize. When considering whether inventing something in-house is a good idea vs using a library, "up front dev cost" factors relatively little to me.
- joquarky 11mo ago
  Don't forget to include supply chain attacks in your risk assessment.
qazxcvbnmlp 11mo ago
Without commenting if parent is right or wrong. (I suspect it is correct) If its true, the market will soon reward it. Being able to competently write good code cheaper will be rewarded. People don't employ programmers because they care about them, they are employed to produce output. If someone can use llms to produce more output for less $$ they will quickly make the people that don't understand the technology less competitive in the workplace.
- zx8080 11mo ago
  > more output for less $$ That's a trap: it's not obvious for those without experience in both business and engineering on how to estimate or later calculate this $$. The trap is in the cost of changes and fix budget when things will break. And things will break. Often. Also, the requirements will change often, that's normal (our world is not static). So the cost has some tendency to change (guess which direction). The thoughtless copy-paste and rewrite-everything approach is nice, but the cost goes up steep with time soon. Those who don't know it will be trapped dead and lose business.
  - tbrownaw 11mo ago
    Predicting costs may be tricky, but measuring them after the fact it's a fair bit easier.
    - zx8080 11mo ago
      Without prediction is like landing B787 totally blind without any instrumental or visual. It will not just hurt, it will kill a business.

Teever 11mo ago

Let's say I'm mildly convinced by your argument. I've read your blog post that was popular on HN a week or so ago and I've made similar little toy programs with AI that scratch a particular niche. Do you care to make any concrete predictions on when most developers will embrace this new normal as part of their day to day routine? One year? Five? And how much of this is just another iteration in the wheel of recarnation[0]? Maybe we're looking at a future where we see return to the monoculture library dense supply chain that we use today but the libraries are made by swarms of AI agents instead and the programmer/user is responsible for guiding other AI agents to create business logic? [0] https://www.computerhope.com/jargon/w/wor.htm https://www.computerhope.com/jargon/w/wor.htm

simonw 11mo ago
It's really hard to predict how other developers are going to work, especially given how resistant a lot of developers are to fully exploring the new tools. I do think there's been a bit of a shift in the last two months, with GPT 5.1 and 5.2 Codex and Opus 4.5. We have models that can reliably follow complex instructions over multiple hour projects now - that's completely new. Those of us at the cutting edge are still coming to terms with the consequences of this (as illustrated by this Karpathy tweet). I don't trust my predictions myself, but I think the next few months are going to see some big changes in terms of what mainstream developers understand these tools as being capable of.
mkozlows 11mo ago
"The future is already here, it's just unevenly distributed." At some companies, most developers already are using it in their day to day. IME, the more senior the developer is, the more likely they are to be heavily using LLMs to write all/most of their code these days. Talking to friends and former coworkers at startups and Big Tech (and my own coworkers, and of course my own experience), this isn't a "someday" thing. People who work at more conservative companies, the kind that don't already have enterprise Cursor/Anthropic/OpenAI agreements, and are maybe still cautiously evaluating Copilot... maybe not so much.

DrammBA 11mo ago

Simon, you're starting to sound super disconnected from reality, this "I hit everything that looks like a nail with my LLM hammer" vibe is new.

simonw 11mo ago
My habits have changed quite a bit with Opus 4.5 in the past month. I need to write about it..
- v64 11mo ago
  Looking forward to hearing about how you're using Opus 4.5, from my experience and what I've heard from others, it's been able to overcome many obstacles that previous iterations stumbled on
- remich 11mo ago
  Please do. I'm trying to help other devs in my company get more out of agentic coding, and I've noticed that not everyone is defaulting to Opus 4.5 or even Codex 5.2, and I'm not always able to give good examples to them for why they should. It would be great to have a blog post to point to…
- indigodaddy 11mo ago
  Can you expound on Opus 4.5 a little? Is it so good that it's basically a superpower now? How does it differ from your previous LLM usage?
  - pertymcpert 11mo ago
    To repeat my other comment: > Opus 4.5 is categorically a much better model from benchmarks and personal experience than Opus 4.1 & Sonnet models. The reason you're seeing a lot of people wax about O4.5 is that it was a real step change in reliable performance. It crossed for me a critical threshold in being able to solve problems by approaching things in systematic ways.
- godelski 11mo ago
  What's concerning to many of us is that you've (and others) have said this same thing s/Opus 4.5/some other model/ That feels more like chasing than a clear line of improvement. It's interrupted very different from something like "my habits have changed quite a bit since reading The Art of Computer Programming". They're categorically different.
  - pertymcpert 11mo ago
    Opus 4.5 is categorically a much better model from benchmarks and personal experience than Opus 4.1 & Sonnet models. The reason you're seeing a lot of people wax about O4.5 is that it was a real step change in reliable performance. It crossed for me a critical threshold in being able to solve problems by approaching things in systematic ways. Why do you use the word "chasing" to describe this? I don't understand. Maybe you should try it and compare it to earlier models to see what people mean.
    - godelski 11mo ago
      > Why do you use the word "chasing" to describe this? I think you'll get the answer to this if you read my comment and your response to understand why you didn't address mine. Btw, I have tried it. It's annoying that people think the problem is not trying. It was getting old when GPT 3.5 came out. Let's update the argument...
  - mkozlows 11mo ago
    It's because the models keep getting better! What you could do with GPT-4 was more impressive than what you could do with GPT 3.5. What you could do with Sonnet 3.5 was more impressive yet, and Sonnet 4, and Sonnet 4.5. Some of these improvements have been minor, some of them have been big enough to feel like step changes. Sonnet 3.7 + Claude Code (they came out at the same time) was a big step change; Opus 4.5 similarly feels like a big step change. (If you don't trust vibes, METR's task completion benchmark shows huge improvements, too.) If you're sincerely trying these models out with the intention of seeing if you can make them work for you, and doing all the things you should do in those cases, then even if you're getting negative results somehow, you need to keep trying, because there will come a point where the negative turns positive for you. If you're someone who's been using them productively for a while now, you need to keep changing how you use them, because what used to work is no longer optimal.
    - godelski 11mo ago
      Models keep getting better but the argument I'm critiquing stays the same. So does the comment I critiqued in the sibling comment to yours. I don't know why it's so hard to believe we just haven't tried. I have a Claude subscription. I'm an ML researcher myself. Trust me, I do try. But that last part also makes me keenly aware of their limitations and failures. Frankly I don't trust experts who aren't critiquing their field. Leave the selling points to the marketing team. The engineer and researcher's job is to be critical. To find problems. I mean how the hell do you solve problems if you're unable to identify them lol. Let the marketing team lead development direction instead? Sounds like a bad way to solve problems > benchmark shows huge improvements Benchmarks are often difficult to interpret. It is really problematic that they got incorporated into marketing. If you don't understand what a benchmark measures, and more importantly, what it doesn't measure, then I promise you that you're misunderstanding what those numbers mean. For METR I think they say a lot right here (emphasis my own) that reinforces my point > Current frontier AIs are vastly better than humans at text prediction and knowledge tasks. They outperform experts on most *exam-style problems* for a fraction of the cost. ... And yet the best AI agents are not currently able to carry out substantive projects by themselves or directly substitute for human labor. *They are unable to reliably handle even relatively low-skill*, computer-based work like remote executive assistance. It is clear that capabilities are increasing very rapidly in some sense, but it is unclear how this corresponds to real-world impact. So make sure you're really careful to understand what is being measured. What improvement actually means. To understand the bounds. It's great that they include longer tasks but also notice the biases and distribution in the human workers. This is important in properly evaluating. Also remember what exactly I quoted. For a long time we've all known that being good at leetcode doesn't make one a good engineer. But it's an easy thing to test and the test correlates with other skills that are likely to be learned to be good at those tests (despite being able to metric hack). We're talking about massive compression machines. That pattern match. Pattern matching tends to get much more difficult as task time increases but this is not a necessary condition. Treat every benchmark adversarialy. If you can't figure out how to metric hack it then you don't know what a benchmark is measuring (and just because you know what can hack it doesn't mean you understand it nor that that's what is being measured)
dimitri-vs 11mo ago
Reality is we went from LLMs as chatbots editing a couple files per request with decent results. To running multiple coding agents in parallel to implement major features based on a spec document and some clarifying questions - in a year. Even IF llms don't get any better there is a mountain of lemons left to squeeze in their current state.

nl 11mo ago
I don't understand what your argument is. It seems to be "people keep saying the models are good"? That's true. They are. And the reason people keep saying it is because the frontier of what they do keeps getting pushed back. Actual, working, useful code completion in the GPT 4 days? Amazing! It could automatically write entire functions for me! The ability to write whole classes and utility programs in the Claude 3.5 days? Amazing! This is like having a junior programmer! And now, with Opus 4.5 or Codex Max or Gemini 3 Pro we can write substantial programs one-shot from a single prompt and they work. Amazing! But now we are beginning to see that programming in 6 months time might look very different to now because these AI system code very differently to us. That's exactly the point. So what is it you are arguing against? I think you said you didn't like that people are saying the same thing, but in this post it seems more complicated?
- timr 11mo ago
  > And now, with Opus 4.5 or Codex Max or Gemini 3 Pro we can write substantial programs one-shot from a single prompt and they work. Amazing! People have been doing this parlor trick with various "substantial" programs [1] since GPT 3. And no, the models aren't better today, unless you're talking about being better at the same kinds of programs. [1] If I have to see one more half-baked demo of a running game or a flight sim...
  - simonw 11mo ago
    "And no, the models aren't better today" Can you expand on that? It doesn't match my experience at all.
    - timr 11mo ago
      It’s a vague statement that I obviously cannot defend in all interpretations, but what I mean is: the performance of models at making non-trivial applications end-to-end, today, is not practically better than it was a few years ago. They’re (probably) better at making toys or one-shotting simple stuff, and they can definitely (sometimes) crank out shitty code for bigger apps that “works”, but they’re just as terrible as ever if you actually understand what quality looks like and care to keep your code from descending into entropy. I think "substantial" is doing a lot of heavy lifting in the sentence I quoted. For example, I’m not going to argue that aspects of the process haven’t improved, or that Claude 4.5 isn't better than GPT 4 at coding, but I still can’t trust any of the things to work on any modestly complex codebase without close supervision, and that is what I understood the broad argument to be about. It's completely irrelevant to me if they slay the benchmarks or make killer one-shot N-body demos, and it's marginally relevant that they have better context windows or now hallucinate 10% less often (in that they're more useful as tools, which I don't dispute at all), but if you want to claim that they're suddenly super-capable robot engineers that I can throw at any "substantial" problem, you have to bring evidence, because that's a claim that defies my day-to-day experience. They're just constantly so full of shit, and that hasn't changed, at all. FWIW, this line of argument usually turns into a mott and bailey fallacy, where someone makes an outrageous claim (e.g. "models have recently gained the ability to operate independently as a senior engineer!"), and when challenged on the hyperbole, retreats to a more reasonable position ("Claude 4.5 is clearly better than GPT 3!"), but with the speculative caveat that "we don't know where things will be in N years". I'm not interested in that kind of speculation.
      - simonw 11mo ago
        Have you spent much time with Codex 5.1 or 5.2 in OpenAI Codex or a Claude Opus 4".5 in Claude code over the last ~6 weeks? I think they represent a meaningful step change in what models can build. For me they are the moment we went from building relatively trivial things unassisted to building quite large and complex system that take multiple hours, often still triggered by a single prompt. Some personal examples from the past few weeks. - A spec-compliant HTML5 parsing library by Codex 5.2: https://simonwillison.net/2025/Dec/15/porting-justhtml/ https://simonwillison.net/2025/Dec/15/porting-justhtml/ - A CLI-based transcript export and publishing tool by Opus 4.5: https://simonwillison.net/2025/Dec/25/claude-code-transcripts/ https://simonwillison.net/2025/Dec/25/claude-code-transcript... - A full JavaScript interpreter in dependency/free Python (!) https://github.com/simonw/micro-javascript https://github.com/simonw/micro-javascript - and here's that transcript published using the above-mentioned tool: https://static.simonwillison.net/static/2025/claude-code-microjs/index.html https://static.simonwillison.net/static/2025/claude-code-mic... - A WebAssembly runtime in Python which I haven't yet published The above projects all took multiple prompts, but were still mostly built by prompting Claude Code for web on my iPhone in between Christmas family things. I have a single-prompt one: - A Datasette plugin that integrates Cloudflare's CAPTCHA system: https://github.com/simonw/datasette-turnstile https://github.com/simonw/datasette-turnstile - transcript: https://gistpreview.github.io/?2d9190335938762f170b0c0eb6060104/index.html https://gistpreview.github.io/?2d9190335938762f170b0c0eb6060... I'm not confident any of these projects would have worked with the coding agents and models we had had four months ago. There is no chance they would've worked with the January 2025 available models.
        timr 11mo ago
        I’ve used Sonnet 4.5 and Codex 5 and 5.1, but not in their native environment [1]. Setting aside the fact that your examples are mostly “replicate this existing thing in language X” [2], again, I’m not saying that the models haven’t gotten better at crapping out code, or that they’re not useful tools. I use them every day. They're great tools, when someone actually intelligent is using them. I also freely concede that they're better tools than a year ago. The devil is (as always) in the details: how many prompts did it take? what exactly did you have to prompt for? how closely did you look at the code? how closely did you test the end result? Remember that I can, with some amount of prompting, generate perfectly acceptable code for a complex, real-world app, using only GPT 4. But even the newest models generate absolute bullshit on a fairly regular basis. So telling me that you did something complex with an unspecified amount of additional prompting is fine, but not particularly responsive to the original claim. [1] Copilot, with a liberal sprinkling of ChatGPT in the web UI. Please don’t engage in “you’re holding it wrong” or "you didn't use the right model" with me - I use enough frontier models on a regular basis to have a good sense of their common failings and happy paths. Also, I am trying to do something other than experiment with models, so if I have to switch environments every day, I’m not doing it. If I have to pay for multiple $200 memberships, I’m not doing it. If they require an exact setup to make them “work”, I am unlikely to do it. Finally, if your entire argument here hinges on a point release of a specific model in the last six weeks…yeah. Not gonna take that seriously, because it's the same exact argument, every six weeks. </caveats> [2] Nothing really wrong with this -- most programming is an iterative exercise of replicating pre-existing things with minor tweaks -- but we're pretty far into the bailey now, I think. The original argument was that you can one-shot a complex application. Now we're in "I can replicate a large pre-existing thing with repeated hand-holding". Fine, and completely within my own envelope for model performance, but not really the original claim.
- pianopatrick 11mo ago
  Is there an endpoint for AI improvement? If we can go from functions to classes to substantial programs then it seems like just a few more steps to rewriting whole software products and putting a lot of existing companies out of business. "AI, I don't like paying for my SAP license, make me a clone with just the features I need".
- godelski 11mo ago
  Two things seem to be in contention: - Models keep getting better[0] - Models since GPT 3 are able to replace junior developers It's true that both of these can be true at the same time but they are still in contention. We're not seeing agents ready to replace mid level engineersand quite frankly I've yet to see a model actually ready to replace juniors. Possibly low end interns but the major utility of interns is to trial run employment. Frankly it still seems like interns and juniors are advancing faster than these models in the type of skills that matter for companies (not to mention that institutional knowledge is quite valuable). But there's interns that started when GPT 3.5 came out that are seniors now. The problem is we've been promised that these employees would be replaced[1] any day now, yet that's not happening. People forget, it is harder to advance when you're already skilled. It's not hard to go from non-programmer to a junior level. Hard to go from junior to senior. And even harder to advance to staff. The difficulty level only increases. This is true for most skills and this is where there's a lot of naivity. We can be advancing faster while the actual capabilities begin to crawl forward rather than leap. [0] Implication is not just at coding test style questions but also in more general coding development. [1] Which has another problem in the pipeline. If you don't have junior devs and are unable to replace both mid and seniors by the time that a junior would advance to a senior then you have built a bubble. There's a lot of big bets being made that this will happen yet the evidence is not pointing that way.
mkozlows 11mo ago
I think you should ask yourself: If it were true that 1) these things do in fact work, 2) these things are in fact getting better... what would people be saying? The answer is: Exactly what we are saying. This is also why people keep suggesting that you need to try them out with a more open mind, or with different techniques: Because we know with absolute first-person iron-clad certainty what is possible, and if you don't think it's possible, you're missing something.

deleted 11mo ago
[deleted]
nl 11mo ago
What do you class a "substantial program"? For me it is something I can describe in a single casual prompt. For example I wrote a fully working version of https://tools.nicklothian.com/llm_comparator.html https://tools.nicklothian.com/llm_comparator.html in a single prompt. I refined it and added features with more prompts, but it worked from the start.
- timr 11mo ago
  Good question. No strict line, and it's always going to be subjective and a little bit silly to categorize, but when I'm debating this argument I'm thinking: a product that does not exist today (obviously many parts of even a novel product will be completely derivative, and that's fine), with multiple views, controllers, and models, and a non-trivial amount of domain-specific business logic. Likely 50k+ lines of code, but obviously that's very hand-wavy and not how I'd differentiate. Think: SaaS application that solves some domain specific problem in corporate accounting, versus "in-browser speadsheet", or "first-person shooter video game with AI, multi-player support, editable levels, networking and high-resolution 3D graphics" vs "flappy bird clone". When you're working on a product of this size, you're probably solving problems like the ones cited by simonw multiple times a week, if not daily.
  - nl 11mo ago
    I don't think anyone is claiming they can one-shot a 50k line SAAS app. I think you'd get close on something like Lovable but that's not really one shot either.
    - nl 11mo ago
      But re-reading your statement you seem to be claiming that there are no 50k SAAS apps that are build even using multi-shot techniques (ie, building a feature at a time). In that case my Vibe-Prolog project would count: https://github.com/nlothian/Vibe-Prolog/ https://github.com/nlothian/Vibe-Prolog/ - It's 45K of python code - It isn't a duplicate of another program (indeed, the reason it isn't finished is because it is stuck between ISO Prolog and SWI Prolog and I need to think about how to resolve this, but I don't know enough Prolog!) - Not a *single* line of code is hand written. Ironically this doesn't really prove that the current frontier models are better because large amounts of code were written with non-frontier models (You can sort of get an idea of what models were used with the labels on https://github.com/nlothian/Vibe-Prolog/pulls?q=is%3Apr+is%3Aclosed https://github.com/nlothian/Vibe-Prolog/pulls?q=is%3Apr+is%3...) But - importantly - this project is what convinced me that the frontier models are much better than the previous generation. There were numerous times I tried the same thing in a non-Frontier model which couldn't do it, and then I'd try it in Claude, Codex or Gemini and it would succeed.
simonw 11mo ago
I know you said don't engage in "you're holding it wrong"... but have you tried these models running in a coding agent tool loop with automatic approvals turned on? Copilot style autocomplete or chatting with a model directly is an entirely different experience from letting the model spend half an hour writing code, running that code and iterating on the result uninterrupted. Here's an example where I sent a prompt at 2:38pm and it churned away for 7 minutes (executing 17 bash commands), then I gave it another prompt and it churned for half an hour and shipped 7 commits with 160 passing tests: https://static.simonwillison.net/static/2025/claude-code-microjs/index.html https://static.simonwillison.net/static/2025/claude-code-mic... I completed most of that project on my phone.
- timr 11mo ago
  > I know you said don't engage in "you're holding it wrong"... but have you tried these models running in a coding agent tool loop with automatic approvals turned on? edit: I wrote a different response here, then I realized we might be talking about different things. Are you asking if I let the agents use tools without my prior approval? I do that for a certain subset of tools (e.g. run tests, do requests, run queries, certain shell commands, even use the browser if possible), but I do not let the agents do branch merges, deploys, etc. I find that the best models are just barely good enough to produce a bad first draft of a multi-file feature (e.g. adding an entirely new controller+view to a web app), and I would never ever consider YOLOing their output to production unless I didn't care at all. I try to get to tests passing clean before even looking at the code. Also, I am happy to let Copilot burn tokens in this manner and will regularly do it for refactors or initial drafts of new features, I'm honestly not sure if the juice is worth the squeeze -- I still typically have to spend substantial time reworking whatever they create, and the revision time required scales with the amount of time they spend spinning. If I had to pay per token, I'd be much more circumspect about this approach.
  - simonw 11mo ago
    Yes, that's what I meant. I wasn't sure if you meant classic tab-based autocomplete or Copilot tool-based agent Copilot. Letting it burn tokens on running tests and refactors (but not letting it merge branches or deploy) is the thing that feels like a huge leap forward to me. We are talking about the same set of capabilities.