3 ms·
A related Dirty Secret that's going to become clear from all this is that a very large proportion of code in the wild (yes, even in 2026—maybe not in FAANG and
by ritlo 7mo ago
A related Dirty Secret that's going to become clear from all this is that a very large proportion of code in the wild (yes, even in 2026—maybe not in FAANG and friends, IDK, but across all code that is written for pay in the entire economy) has limited or no automated test coverage, and is often being written with only a limited recorded spec that's usually fleshed out only to the degree needed (very partial) as a given feature is being worked on.
What do the relatively hands-off "it can do whole features at a time" coding systems need to function without taking up a shitload of time in reviews? Great automated test coverage, and extensive specs.
I think we're going to find there's very little time-savings to be had for most real-world software projects from heavy application of LLMs, because the time will just go into tests that wouldn't otherwise have been written, and much more detailed specs that otherwise never would have been generated. I guess the bright-side take of this is that we may end up with better-tested and better-specified software? Though so very much of the industry is used to skipping those parts, and especially the less-capable (so far as software goes) orgs that really need the help and the relative amateurs and non-software-professionals that some hope will be able to become extremely productive with these tools, that I'm not sure we'll manage to drag processes & practices to where they need to be to get the most out of LLM coding tools anyway. Especially if the benefit to companies is "you will have better tests for... about the same amount of software as you'd have written without LLMs".
We may end up stuck at "it's very-aggressive autocomplete" as far as LLMs' useful role in them, for most projects, indefinitely.
On the plus side for "AI" companies, low-code solutions are still big business even though they usually fail to deliver the benefits the buyer hopes for, so there's likely a good deal of money to be made selling companies LLM solutions that end up not really being all that great.
- _wire_ 7mo ago> because the time will just go into tests that wouldn't otherwise have been written Writing tests to ensure a program is correct is the same problem as writing a correct program. Evaluating conformance is a different category of concern from ensuring correctness. Tests are about conformance not correctness. Ensuring correct programs is like cleaning in the sense that you can only push dirt around, you can't get rid of it. You can push uncertainty around and but you can't eliminate it. This is the point of Gödel's theorem. Shannon's information theory observes similar aspects for fidelity in communication. As Douglas Adams noted: ultimately you've got to know where your towel is.
- deleted 7mo ago[deleted]
- layer8 7mo agoA competent programmer proves the program he writes correct in his head. He can certainly make mistakes in that, but it’s very different from writing tests, because proofs abstract (or quantify) over all states and inputs, which tests cannot do.
- slopinthebag 7mo agoRe. productivity, if LLM's are a genuine boost with 1/3 of the work, neutral 1/3 of the time, and actually worse 1/3 of the time, it's likely we aren't really seeing performance improvements as 1) people are using them for everything and b) we're still learning how to best use them. So I expect over time we will see genuine performance improvements, but Amdahl's law dictates it won't be as much as some people and ceo's are expecting.
- ansibsha 7mo ago> better-specified software Code is the most precise specification we have for interfacing with computers.
- tmaly 7mo agoThere are some cases where AI is generating binary machine code, albeit small amounts. What do we have when we don't have the code?
- marginalia_nu 7mo agoMachine code is still code, even if the representation is a bit less legible than the punch cards we used to use.
- deleted 7mo ago[deleted]
- interestpiqued 7mo agoYou’re missing the point of a spec
- unselect5917 7mo agoThe spec is as much for humans as it is the machine, yes?
- interestpiqued 7mo agoSpec should be made before hand and agreed on by stakeholders. It says what it should do. So it’s for whoever is implementing, modifying, and/or testing the code. And unfortunately devs have a tendency of poor documentation
- mrguyorama 7mo agoSoftware development is only 70ish years old and somehow we have already forgotten the very very first thing we learned. "Just get bulletproof specs that everyone agrees on" is why waterfall style software development doesn't work. Now suddenly that LLMs are doing the coding, everyone believes that changes?
- dboreham 7mo agoBingo. Hopefully there are some business opportunities for us in that truth.