5 ms·
There's no chance LLMs will be an engineering team replacement. The hallucination problem is unsolvable and catastrophic in some edge cases. Any company using s
by smt88 10mo ago
There's no chance LLMs will be an engineering team replacement. The hallucination problem is unsolvable and catastrophic in some edge cases. Any company using such a team would be uninsurable and sued into oblivion.
- eru 10mo agoWriting software is actually one of the domains where hallucinations are easiest to fix: you can easily check whether it builds and passes tests. If you want to go further, you can even require the LLM to produce a machine checkable proof that the software is correct. That's beyond the state of the art at the moment, but it's far from 'unsolvable'. If you hallucinate such a proof, it'll just not work. Feed back the error message from the proof checker to your coding assistant, and the hallucination goes away / isn't a problem.
- ulrikrasmussen 10mo agoIn order to prove safety you need a formal model of the system and formally defined safety properties that are both meaningful and understandable by humans. These do not exist for enterprise systems
- eru 10mo agoAn exhaustive formal spec doesn't exist. But you can conservatively proof some properties. Eg program termination is far from sufficient for your program to do what you want, but it's probably necessary. (Termination in the wider sense: for example an event loop has to be able to finish each run through the loop in finite time.) You can see eg Rust's or Haskell's type system as another light-weight formal model that lets you make and proof some simple statements, without having a full formal spec of the whole desired behaviour of the system.
- tsimionescu 10mo agoThat is true and very useful for software development, but it doesn't help if the goal is to remove human programmers from the loop entirely. If I'm a PM who is trying to get a program to, say, catalogue books according to the Dewey Decimal system for a library, a proof that the program terminates is not going to help that much when the program is mis-categorizing some books.
- seanmcdirmid 10mo agoIs removing the human in the loop really the goal, or is the goal right now to make the human a lot more productive? Because...those are both very different things.
- tsimionescu 10mo agoI don't know what the goal for OpenAI or Anthropic really is. But the context of this thread is the idea that the user daxfohl launched that these companies will, in the next few years, launch an "engineering team replacement" program; and then the user eru claimed that this is indeed more doable in programming than other domains because you can have specs and tests for programs in a way that you can't for, say, an animated movie.
- eru 10mo agoOK, so you successfully argued that replacing the entire engineering team is hard. But you can perhaps still shrink it by 99%. To the point where a sole founder can do the remaining tech role part time.
- seanmcdirmid 10mo agoI have no idea what will happen in a few years, maybe LLM tech will hit a wall and humans will continue to be needed in the loop. But today humans are definitely needed in the loop in some way.
- ulrikrasmussen 10mo agoYeah, but with all respect, that is a totally uninteresting property in an enterprise software system where almost no software bugs actually manifest as non-termination. The critical bugs here are related to security (DDoS attacks, authorization and authentication, data exfiltration, etc), concurrency, performance, data corruption, transactionality and so forth. Most enterprise systems are distributed or at least concurrent systems which depend on several components like databases, distributed lock managers, transaction managers, and so forth, where developing a proper formal spec is a monumental task and possibly impossible to do in a meaningful way because these systems were not initially developed with formal verification in mind. The formal spec, if faithful, will have to be huge to capture all the weird edge cases. Even if you had all that, you need to actually formulate important properties of your application in a formal language. I have no idea how to even begin doing that for the vast majority of the work I do. Proving the correctness of linear programs using techniques such as Hoare logic is hard enough already for anything but small algorithms. Proving the correctness of concurrent programs operating on complex data structures requires much more advanced techniques, setting up complicated logical relations and dealing with things like separation logic. It's an entirely different beast, and I honestly do not see LLMs as a panacea that will suddenly make these things scale for anything remotely close in size to a modern enterprise system.
- eru 10mo agoOh, there's lots more simple properties you can state and prove that capture a lot more, even in the challenging enterprise setting. I just gave the simplest example I could think of. And termination is actually a much stronger and more useful property than you make it out to be---in the face of locks and concurrency.
- solid_fuel 10mo ago> Writing software is actually one of the domains where hallucinations are easiest to fix: you can easily check whether it builds and passes tests. What tests? You can't trust the tests that the LLM writes, and if you can write detailed tests yourself you might as well write the damn software.
- eru 10mo agoUse multiple competing LLM. Generative adversarial network style.
- solid_fuel 10mo agoCool. That sure sounds nice and simple. What do you do when the multiple LLMs disagree on what the correct tests are? Do you sit down and compare 5 different diffs to see which have the tests you actually want? That sure sounds like a task you would need an actual programmer for. At some point a human has to actually use their brain to decide what the actual goals of a given task are. That person needs to be a domain expert to draw the lines correctly. There's no shortcut around that, and throwing more stochastic parrots at it doesn't help.
- eru 10mo agoJust because you can't (yet) remove the human entirely from the loop, doesn't mean that economising on the use of the humans time is impossible. For comparison have a look at compilers: nowadays approximately no one writes their software by hand, we write a 'prompt' in something like Rust or C, and ask another computer program to create the actual software. We still need the human in the loop here, but it takes much less human time than creating the ELF directly.
- exceptione 10mo agoI can see how LLMs can help with testing, but one should never compare LLMs with deterministic tools like compilers. LLMs are entirely a separate category.
- DrammBA 10mo agoYou focused on writing software, but the real problem is the spec used to produce the software, LLMs will happily hallucinate reasonable but unintended specs, and the checker won’t save you because after all the software created is correct w.r.t. spec. Also tests and proof checkers only catch what they’re asked to check, if the LLM misunderstands intent but produces a consistent implementation+proof, everything “passes” and is still wrong.
- simonw 10mo agoThis is why every one of my coding agent sessions starts with "... write a detailed spec in spec.md and wait for me to approve it". Then I review the spec, then I tell it "implement with red/green TDD".
- daxfohl 10mo agoSame, and similarly something like a "create a holistic design with all existing functionality you see in tests and docs plus new feature X, from scratch", then "compare that to the existing implementation and identify opportunities for improvement, ranked by impact, and a plan to implement them" when the code starts getting too branchy. (aka "first make the change easy, then make the easy change"). Just prompting "clean this code up" rarely gets beyond dumb mechanical changes. Given so much of the work of managing these systems has become so rote now, my only conclusion is that all that's left (before getting to 95+% engineer replacement) is an "agent engineering" problem, not an AI research problem.
- tsimionescu 10mo agoThe premise was that the AI solution would replace the engineering team, so who exactly is writing/reviewing this detailed spec?
- simonw 10mo agoThat's a bad premise.
- 10mo ago
- mohaine 10mo agoAh, most the problem in programming is writing the tests. Once you know what you need the rest is just typing. I can see an argument where you can get none programers to create the input and output of said tests but if the can do that, they are basically programmers. This is of course leaving aside that half the stated use cases I hear for AI are that it can 'write the tests for you'. If it is writing the code and the tests it is pointless.
- discreteevent 10mo agoYou need more than tests. Test induced design damage: https://dhh.dk/2014/test-induced-design-damage.html https://dhh.dk/2014/test-induced-design-damage.html
- somenameforme 10mo agoTests and proofs can only detect issues that you design them to detect. LLMs and other people are remarkably effective at finding all sorts of new bugs you never even thought to test against. Proofs are particularly fragile as they tend to rely on pre/post conditions with clean deterministic processing, but the whole concept just breaks down in practice pretty quickly when you start expanding what's going on in between those, and then there's multithreading...
- thesz 10mo ago> you can easily check whether it builds and passes tests. This link were on HN recently: https://spectrum.ieee.org/ai-coding-degrades https://spectrum.ieee.org/ai-coding-degrades "...recently released LLMs, such as GPT-5, have a much more insidious method of failure. They often generate code that fails to perform as intended, but which on the surface seems to run successfully, avoiding syntax errors or obvious crashes. It does this by removing safety checks, or by creating fake output that matches the desired format, or through a variety of other techniques to avoid crashing during execution." The trend for LLM generated code is to build and pass tests but do not deliver functionality needed. Also, please consider how SQLite is tested: https://sqlite.org/testing.html https://sqlite.org/testing.html The ratio between test code and code itself is mere 590 times (590 LOC of tests per LOC of actual code), it used to be more than 1100. Here is notes on current release: https://sqlite.org/releaselog/3_51_2.html https://sqlite.org/releaselog/3_51_2.html Notice fixes there. Despite being one of the most, if not the most, tested pieces of software in the world, it still contains errors. > If you want to go further, you can even require the LLM to produce a machine checkable proof that the software is correct. Haha. How do you reconcile a proof with actual code?
- eru 10mo ago> Haha. How do you reconcile a proof with actual code? You can either proof your Rust code correct, or you can use a proof system that allows you to extract executable code from the proofs. Both approaches have been done in practice. Or what do you mean?
- thesz 10mo agoRust code can have arbitrary I/O effects in any parts of it. This precludes using only Rust's type system to make sure code does what spec said. The most successful formally proven project I know, seL4 [1], did not extracted executable code from the proof. They created a prototype in Haskell, mapped (by hand) it to Isabelle, I believe, to have a formal proof and then recreated code in C, again, manually. [1] https://sel4.systems/ https://sel4.systems/ Not many formal proof systems can extract executable C source.
- 10mo ago
- shevy-java 10mo agoWell - the end result can be garbage still. To be fair: humans also write a lot of garbage. I think in general most software is rather poorly written; only a tiny percentage is of epic prowess.
- rezonant 10mo agoWho writes the tests?
- Marazan 10mo agoWho is writing the tests?
- fragmede 10mo agoI am but a lowly IC, with no notion of the business side of things. If I am an IC at, say, a FANG company, what insurance has been taken out on me writing code there?
- smt88 10mo ago> If I am an IC at, say, a FANG company, what insurance has been taken out on me writing code there? Every non-trivial software business has liability insurance to cover them for coding lapses that lead to data breaches or other kinds of damages to customers/users.
- kristiandupont 10mo agoI use LLM's to write the majority of my code. I haven't encountered a hallucination for the better part of a year. It might be theoretically unsolvable but it certainly doesn't seem like a real problem to me.
- deleted 10mo ago[deleted]
- smt88 10mo agoI use LLMs whenever I'm coding, and it makes mistakes ~80% of the time. If you haven't seen it make a huge mistake, you may not be experienced enough to catch them.
- kristiandupont 10mo agoHallucinations, no. Mistakes, yes, of course. That's a matter of prompting.
- tdrz 10mo ago> That's a matter of prompting. So when I introduce a bug it's the PM's fault.
- matwood 10mo agoThese types of comments are interesting to me. Pre-chatGPT there were tons of posts how so many software people were terrible at their jobs. Bugs were/are rampant. Software bugs caused high profile issues, but likely so many more we never heard about. Today we have chatGPT and only now will teams be uninsurable and sued into oblivion? LOL
- elzbardico 10mo agoLLMs were trained on exactly that kind of code.
- smt88 10mo agoIf you've ever used Claude Code in brave mode, I can't understand how you'd think a dev team could make the same categories of mistakes or with the same frequency.