9 ms·

> Writing non-trivial software that is correct (for any meaningful definition of correct) is beyond the current capabilities of the human species. Bullshit. Hu

> Writing non-trivial software that is correct (for any meaningful definition of correct) is beyond the current capabilities of the human species. Bullshit. Humans have been sending computers into space for decades, and while yes, some have had programming problems, most worked correctly, for a very specific definition of correctly. Thing is, NASA (and Russian and Chinese and Indian and ...) space engineers have approached their challenges from the point of view of actual engineering, while modern "software engineering" is all about craftsmanship, and just a sprinkle of actual engineering to make ourselves feel good.

veeralpatel979 7y ago
Writing "correct" software is possible IMO, but is it worth it? For most applications, is it worth spending several times more time and resources in order to make it "correct"? I don't think so.
- mristin 7y ago
  I always wondered where this multiplicative factor of "several times" comes from. In my experience, writing correct software was marginally slower than writing sloppy software as long as most thinking is done with a pen & paper. Would you mind to elaborate a bit more?
  - TeMPOraL 7y ago
    I'm guessing: because of cheap labor. Writing correct and/or fast software involves quite a bit of "don't do stupid things" all across the development process. Doing these things right doesn't add much time to development, but one has to first learn how to do these things right. A fresh and inexperienced developer, or the "one year of experience repeated 10 times" person that only worked at "move fast and break things" project isn't going to have this knowledge, but will be cheaper to hire.
  - beagle3 7y ago
    Not OP/GP, but I think it's mostly about the definition of correct. Does it correctly handle every possible sequence of inputs? For the vast majority of software in use today, the answer is "no"; The follow up question is "does it matter?" and (luckily or unluckily) the answer for the vast majority of software in the vast majority of use cases, is "no" as well. But occasionally it does, e.g. https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 is a famous poster child, and I quote: >>> The failure occurred only when a particular nonstandard sequence of keystrokes was entered on the VT-100 terminal which controlled the PDP-11 computer: an "X" to (erroneously) select 25 MeV photon mode followed by "cursor up", "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds Software was not obviously correct or incorrect; In fact, it had been acceptable for an earlier model (which had some hardware protections missing from the newer one). Reaching the incorrect state required the race described above to trigger, which did, in fact, happen in practice a handful of times. You can work very hard to formally prove your implementation, only to find out that the compiler had a bug and makes your software bad. Or the CPU does; Or all the designs are fine, but there's an bit flip due to electro-migration or cosmic rays. Many people consider this "force majeur" - "an act of god" one cannot anticipate, but cosmic rays are in fact an expected -- and hard to avoid -- input to many systems. You are in control of a logical model which you can, with extra work, do (provably) correctly. But that IS, from experiencce, 10x to 100x more expensive, and unless you go for 10x-100x more expensive hardware, reduces and moves the sloppiness factor around, but does not eliminate it.
- mojuba 7y ago
  Depends on what consequences your software can have in the real world. Most of software that you run on the laptop can glitch and crash all it wants as long as your users can tolerate that, but software for cars for example can have very real consequences on lives. We should be worried about poor development practices in the automotive industry specifically. Looks like many manufacturers approach the tasks the same way any other hardware manufacturer does, i.e. software is secondary in their minds. Software for a car is kind of the same as software for your next "smart fridge". Among other things, it can be outsourced for example. Whereas, as Bruce Schneier put it, a modern car is a computer on wheels, not the other way around. A computer on wheels is a lot more dangerous (and there are security implications too, it's what Schneier meant actually).
- TeMPOraL 7y ago
  It's a complex issue, because often the answer "it's not worth it" is based on company's ability to make someone else pay for the consequences.
- ehnto 7y ago
  How far down the stack does it need to be correct? Can a 100% correct Python script be considered correct if there could still be bugs further down the stack? Also what's the context of the software? Much software has pretty safe failure modes, but if it's a medical system for example, knowing level of correctness right down to hardware level may be useful. I think in that light, most modern software can be built to be robust and easily maintainable rather than correct. Like the difference between building a bridge, where you understand everything in the system, versus building a vehicle engine, where you have multiple dependencies but you still need it to not catastrophically fail should any of them stop being correct (eg: fuel pump fails, engine management detects lower fuel pressure and cuts ignition, saving the engine).
- LandR 7y ago
  Everyone seems to think that, but I feel this ends up being death by a thousand cuts. I don't know a single piece of software on my computer right now that doesn't crash, act weird, hang or do something else to annoy me on a daily basis. Now yes, each infraction might be just a minor niggle, but god it adds up! By the end of the day, all the bugs that on their own shouldn't be too bad, leave me frustrated as hell and wanting to say fuck everyone and just go live in the woods. It's especially worse when I know this shit doesn't need to happen.
  - LargeWu 7y ago
    Compare that to the cost of software that works perfectly, but costs 10x what it currently does in order to achieve that perfection. Think of all the free (gratis) software you use that maybe would have a huge cost associated with it to be as reliable as you would like. It's all about tradeoffs. Everybody wants cheap software that works perfectly, but reality dictates that can't happen. Most of the time, cheap/free with minor annoyances is better than perfect but high-cost.
bilekas 7y ago
I came to write something simular.. Just because you havent come accross it, doesn't mean its beyond our capabilities, infact, by definition its within our means.
scoutt 7y ago
Yeah, I think the author is being a little pessimistic here, but for the benefit of the doubt it may be interesting to understand what correct means in that sentence. I mean, if correct = 100% uptime, 100% effectiveness, etc., then it's not so much of a software problem, but a hardware one.
- mrkeen 7y ago
  Yeah but software fails so often and repeatedly that it's rarely worth asking the question "is it software or hardware?"
  - scoutt 7y ago
    Exploding batteries, bad wiring, design faults, silicon bugs and erratas, bad connectors, temperature problems... I've seen them all. It's more common than one may usually think, especially if you work in embedded. And somehow I wanted to tie the answer with the concept of the parent about space equipment. BTW, "is it software or hardware?" is exactly what the boss asks first when a customer has a problem.
    - ehnto 7y ago
      Regarding my reply to your parent comment, it sounds like we are both biased by our experience, we might need some data!
      - scoutt 7y ago
        I agreed about the bias. When I read "software" I try to think about a broad spectrum of software applications, including the ones that are running in your fridge, router or the ISS. When author (or most people, I believe) mentions software, it seems it's just apps (like food apps mentioned elsewhere) or web apps, that is just a fraction.
  - jonathanstrange 7y ago
    I wouldn't be so sure about that. My PC has an overclocked AMD processor with 32GB non-ECC RAM (because ECC ram wasn't even available when I bought it, let alone affordable). Even if the software was 100% perfect, such a machine is expected to crash once or twice every year or so under full load just from the failure rate of the chips (cosmic radiation / quantum tunnelling effects). Especially RAM seems to have become fairly error-prone nowadays.
onei 7y ago
It's kind of curious coming out of university where I learned about doing requirements properly and up front (which I suspect kind of tends towards a waterfall approach) then comparing that approach to my work now. Project managers constantly change their mind about requirements and then wonder why we're shipping late, although I suspect some of this is not considering failure/bugs when planning. Even when it comes to breaking down and estimating work, if the estimates software engineers produce don't align with the high level schedule they're automatically wrong or people start asking about what we can trim down (which somehow is always testing). Perhaps it's idealism vs reality, but I find it a bit bizarre.
- szatkus 7y ago
  Because it's much easier to change software than for example a bridge. I don't see it as a bad thing as long as all sides are aware that changing requirements changes the deadline as well.
  - reallydontask 7y ago
    > I don't see it as a bad thing as long as all sides are aware that changing requirements changes the deadline as well. In my experience, this could not be further from the truth. There is simply no awareness that change remains expensive it's just that now the process allows it without any fanfare whereas before a change was a big deal. Not saying that change requests are the way to go but certainly an understanding that change is not free would go a long way
    - iainmerrick 7y ago
      Change in a software project definitely isn’t free but I do think it’s fair to say it’s easier than changing a bridge design halfway through. Maybe there’s not as big a difference as people think, though? It occurs to me that is actually possible to change the design of a bridge late in the day. The extra dampers to fix resonance problems on London’s Millennium Bridge are a nice example.
      - 7y ago
jerome-jh 7y ago
It is said that the Apollo 11 lunar lander computer rebooted twice during the final approach. We could probably not say it was correct, but still it was robust. "for any meaningful definition of correct" sounds a bit to strong as an hypothesis. Maybe we could rephrase: for all piece of software S, there exist a definition of correctness for which S is incorrect. Timings in particular are very difficult to ensure.
- fjfaase 7y ago
  I did not reboot. (I think it did not have the capability to reboot itself.) The program alarms indicated "executive overflows", meaning the guidance computer could not complete all its tasks in real time and had to postpone some of them. https://en.wikipedia.org/wiki/Apollo_11#Lunar_descent https://en.wikipedia.org/wiki/Apollo_11#Lunar_descent See also http://klabs.org/history/apollo_11_alarms/eyles_2004/eyles_2004.htm http://klabs.org/history/apollo_11_alarms/eyles_2004/eyles_2... for a detailed explanation.
  - Izkata 7y ago
    > I did not reboot. ...Got something to share with us? ;)
adrianN 7y ago
Writing non-trivial, correct software while not blowing the budget is beyond the capabilities of the human species.
- pif 7y ago
  Allocating the correct budget to write non-trivial, correct sooftware is beyond the capabilities of idiot managers.
  - adrianN 7y ago
    I'm pretty sure almost nobody wants to pay millions of dollars for a copy of Word that never crashes.
    - diafygi 7y ago
      But will they pay millions of dollars for a WiFi firmware stack in energy IoT devices that the grid is increasingly depending on that isn't vulnerable to memory overflows or other hacking vectors? Software is becoming more and more depended on for life and death use cases every day.
      - adrianN 7y ago
        Life-and-death software is already regulated fairly strictly and generally has decent quality. But of course the companies developing it also try to cut costs and in the end it's more about checking off boxes to avoid liability than producing correct software.
        diafygi 7y ago
        Hmmm, on the sliding scale of harmless to life-and-death software, it seems that as time goes many programs and services migrate from being closer to the harmless end to bring closer to the life-and-death end. I feel that migration is often ignored or discounted. For example, Facebook in the early years was considered mostly harmless, but now has migrated to being exploited by state actors to brainwash populations into hating each other, interfering in elections, or at worst performing genocide on a minority group. We need to stop assuming that just because a software application is harmless now, that it will stay that way, and we need to adjust its "correctness" accordingly as it migrates along the harmless <> life-and-death scale.
yitchelle 7y ago
The effort needed to create a "hello, world" program is not trivial. ie, creation of the tool chain (compiler, linker), the OS which to run the "hello, world executable", the text editor to write the code, the code that is in your keyboard, and the list goes on. All software. No software is trivial if the author wants to broaden his argument to the human species.
tn890 7y ago
>Thing is, NASA (and Russian and Chinese and Indian and ...) space engineers have approached their challenges from the point of view of actual engineering, while modern "software engineering" is all about craftsmanship, and just a sprinkle of actual engineering to make ourselves feel good. What would be the quintessential differences here? We plan, we test, we revise. I'd argue we're all doing the same thing, but if you're working on a food ordering app you can afford not to be as stringent with the testing and quality, versus software that takes people into outer space. It's just a matter of stronger concerns. Things matter more in a field where you can kill people if you have a bug, so everyone strives to do a better job, top to bottom.
- imgabe 7y ago
  The difference is measuring and quantification. Engineering means you know, or can estimate, a load and perform calculations to see if your design meets that load. Most food order apps seem to be more like, eyeball it, slap something together, then it works until it doesn't, at which point you figure out a fix. That works for a while until something else breaks, so you patch a fix into that. Ad infinitum.
  - BigJono 7y ago
    Funny you mention food order apps. UberEats web app is my prime example of how a flaming train wreck can make you a ludicrous amount of money, and how there's almost no correlation between engineering quality (not necessarily product quality) and revenue.
    - exdsq 7y ago
      > there's almost no correlation between engineering quality (not necessarily product quality) and revenue. Agreed. Sadly I'm much more interested in engineering quality than revenue..
    - tatersolid 7y ago
      Uber loses fantastic, mind-bending amounts of money every quarter. Their apps don’t “make money.”
raverbashing 7y ago
There's a very nice graph on "The Mythical Man-Month" about this. Basically, if you constrain your software to work for your specific use case, in your specific system, it's "100x" easier. Also you don't know how many bugs, known or unknown exist in software on space probes.
onion2k 7y ago
most worked correctly, for a very specific definition of correctly. That doesn't mean incorrectness isn't there. That means the circumstances necessary for any incorrectness to manifest in a measurable way didn't happen.
- WJW 7y ago
  Perhaps, but that is true for any form of engineering. There is no building on earth that can withstand impacts by Texas-sized meteorites, making them incorrectly constructed in those very specific circumstances. Correctness is whether it performs to the requirements of the spec, and if that spec contains tradeoffs that accept non-functioning in certain extreme circumstances then not functioning in those circumstances is NOT incorrect behavior.
  - onion2k 7y ago
    I'm not talking about extraordinary events like a giant meteor strike though. I strongly suspect there are plenty of buildings that would collapse if there was a small flood or if they were hit by a car in the wrong place when they have been designed to remain standing in those circumstances. They're incorrectly designed, or incorrectly built, or incorrectly maintained. There's so many ways for things to fail. The only reason they've not collapsed is because those circumstances have never arisen. They probably never will. That doesn't make them correct.
PopeDotNinja 7y ago
Software that doesn't have changing requirements is eventually trivial. You can logic your way to a reasonable solution eventually. Building an app that constantly evolves is never trivial.
- exdsq 7y ago
  Software without changing requirements is absolutely not eventually trivial, unless ‘eventually’ is a nice word to mask all of the required project complexity. The Mars Rover has specific requirements, is this more trivial than a startups new web-app?
makach 7y ago
I am always amazed when I think about how many instructions a computer goes through every time it does a cold-start and the first pixel is shown on a screen. I particularly liked that you wrote "most worked correctly".
LandR 7y ago
Most software is bad because most developers are bad. I don't think it's more complicated than that.
- exdsq 7y ago
  Not just that but the discipline is bad because it’s young. The fact that the majority of IT projects fail is just crazy.
- zeroxfe 7y ago
  Good developers can write bad software too. Primarily because there are a lot more factors to software quality than just developer quality: management quality, team dynamics, cost/time pressure, market dynamics, etc.
lpilot 7y ago
I kind of see where they're coming from with this one, although I don't think it's true. The definitions of trivial and correct can be moved to fit whatever they're trying to say. Rephrasing it as something like it's incredibly difficult to write a large piece of software with no bugs makes more sense. And I think for most organisations it may as well be impossible.
deleted 7y ago
[deleted]
bartread 7y ago
> Thing is, NASA (and Russian and Chinese and Indian and ...) space engineers have approached their challenges from the point of view of actual engineering, while modern "software engineering" is all about craftsmanship, and just a sprinkle of actual engineering to make ourselves feel good. You are, of course, absolutely correct. Context is key here: you treat software engineering like "real" engineering when it's appropriate to do so (space flight, aircraft control systems, nuclear power plant monitoring and control, medical devices, and so forth). However, I think for many organisations - and for much of the time - the engineering aspect is often heavily overplayed, and actually you're trading that off against two key aspects of software that can be used to build a real competitive advantage. Those being (i) speed of implementation, and (ii) malleability. Building software can get you to a solution very quickly. You can also iterate or modify that solution very quickly. Clearly there are limits to this but for many companies, and most startups, getting something working quickly, and iterating on it quickly is far more important than any engineering merit. Same goes for larger companies introducing a new product: don't overegg the "engineering" because it will slow you down and time may be the only sustainable competitive advantage you have. There's a time to be an engineer, and there's a time to be a cowboy hacker (and everything in between). The real trick is in knowing where on the spectrum to execute in the context of the customer base you're serving and the software you're building.
cryptica 7y ago
Even the NASA curiosity rover encountered a software error and now it's stuck. I think OP's argument stands. And NASA software is not that complex if you think about it; at least it has fixed requirements; once they ship it, it's done. No more changes. This is different from some complex software which require constant changes and adaptation to changing requirements; in this case the code has to be designed to adapt easily; this is difficult to achieve and most developers/companies fail at that or in the case of companies, they only just barely succeed through the development of extremely expensive solutions. Big companies like Google don't write efficient software; they use their capital to hire as many expensive engineers as possible to brute force the requirements until it works. That's why they keep rewriting projects from scratch every few years over and over; they don't know how to write adaptable software. Their software is somewhat complex, but still simple enough and they have enough capital so that brute forcing solutions are still feasible (for a high cost). The same cannot be said about more complex tech like blockchain or large-scale machine learning systems; you can't brute force your way to a solution and full rewrites will actually set you back.
- exdsq 7y ago
  > And NASA software is not that complex if you think about it; at least it has fixed requirements; once they ship it, it's done. No more changes. Nope, they still make changes. This is a case study for a company that helped debug and fix a low-level C bug on the Mars Curiosity Rover as it was traveling to Mars. https://semmle.com/case-studies/semmle-nasa-landing-curiosity-safely-mars https://semmle.com/case-studies/semmle-nasa-landing-curiosit... I work with high-assurance software in the blockchain space and only recently read a paragraph from some old NASA software quality textbook and thought damn.. this is awesome. It's anything but trivial!
arexxbifs 7y ago
I've seen the claim before that it was better in the olden days and that "no real engineer" would create software bugs, especially in connection with space flight. Meanwhile, here is a list of space flight software bugs that has cost over a billion dollars and put human lives in jeopardy: https://en.wikipedia.org/wiki/List_of_software_bugs#Space https://en.wikipedia.org/wiki/List_of_software_bugs#Space
- jonathanstrange 7y ago
  > I've seen the claim before that it was better in the olden days and that "no real engineer" would create software bugs But who would claim that seriously? That would be obvious hyperbole. The claim is rather that certain teams of engineers have created software/hardware combinations that, for all we know and for all practical purposes, did not have a bug and have not failed due to software error. The development of high integrity systems is costly, though, so the discussion is a bit moot. Sure it would be possible to develop an Android app that - within the limitations of those devices and the buggy operating systems - would not fail due to a bug in its own program. It's just really expensive, particularly if the software is also formally verified. Generally speaking, it doesn't even make sense to consider "high integrity software" without the accompanying hardware. You can only satisfy real-time constraints for specific hardware anyway, and the certification and evaluation should be for software+hardware.
  - arexxbifs 7y ago
    I honestly think the post I commented on more or less made this claim by calling the original statement "bullshit", writing off billion-dollar crashes and real risk of death as "programming problems" and calling it "actual engineering". All humans are capable of unkowingly making mistakes which can have dire consequences. We've yet to see an approach to software development that will, without a shadow of a doubt, erradicate the possibility of bugs.
sacado2 7y ago
And it's obviously not the developers' fault, but the clients/customers'. You want a 100% bug-free web browser? OK, so let's define precisely and formally what the browser is supposed to do. Then, you'll sign a very expensive contract, and in 6 months to 1 year, I'll deliver the software, with a formal proof it works as expected. Of course, the requirements won't change during the development process, and any update after delivery will be costly and will take time. Now, who would want to pay thousands of dollars for a web browser that will be delivered in a year and that will be obsolete as soon as a new fancy web tech will be used everywhere else? I can't really blame the customer/client either. But the thing is, "agile" requirements rely on craftmanship, not on engineering. You can only engineer durable things. Buildings would crash constantly, too, if they had to be updated each other week, ASAP, and while keeping the costs as low as possible.
- 6510 7y ago
  What fun proposal! No one wants it but it is exactly what we need. It should do a bunch of really creative things then further new fancy web tech shouldn't exist. Something like: Distribute documents signed by their author in a truly robust way. It should have A review and certification system and It should help monetize publishing in a truly transparent way. Write it in hardware design language and run it in an emulator. :-)
- C1sc0cat 7y ago
  In the early days of the web 94 I worked on web project for British Telecom. I was told that some one had said well this protocol is all very well but we need to build our own slightly different improved version. Luckily this was stomped on from a great Hight by senior people at BT Labs
- TheCoelacanth 7y ago
  I think you're underestimating the amount of work involved in creating a web browser by multiple orders of magnitude. If it really took a year to make a bug-free browser, we would have 100 of them instead of 0.
- takeda 7y ago
  If you want to see extreme cases of this look at code of games. It's no wonder they are so buggy when released.
clarry 7y ago
I think correctness is achievable for a lot of things, but the industry at large isn't chasing it. People ask for a feature. How long does it take, how much does it cost? Dang, can we get it sooner? Nobody ever asked me about what it takes to ensure it is correct. Nobody asked me to work with them to produce a detailed specification that exhaustively considers all cases that must be accounted for, nobody asked for a formal proof, nobody asked me to run a model checker, not even do something like design by contract. And when I look at job postings, all this stuff is completely absent. I was excited when I found one company that recognized Ada and Spark, and got me into an interview for mentioning them. Just sprinkle some assert and make a few tests. If shit stick to ceiling, it's cooked. That's how the industry works. We aren't even trying. It's pretty funny that everyone's making claims about what's possible when nobody is trying. Yep, I bet there are niches, like in aerospace, where some try, and I'm sure some of them also succeed at correctness. We only hear about the failures.
- ryandrake 7y ago
  Exactly. Correctness (and often mere robustness) is almost never a requirement, so it's never even attempted. There are industries where it is important and where it is done routinely, but evidently not in the places most HN commenters work. To say it's impossible probably sounds kind of silly to the engineers who are currently doing it.
  - crimsonalucard 7y ago
    Most HN commenters don't even know that correctness is possible. They don't even know about correctness without testing, they tie correctness and testing together and think it's the same thing. >There are industries where it is important and where it is done routinely, but evidently not in the places most HN commenters work. Can you name these industries?
0x445442 7y ago
Couldn't agree more but I can say it wasn't just NASA. The industry as a whole used to be more aligned with engineering but this has changed considerably since, say, the rise of "Agile".
- cr0sh 7y ago
  > The industry as a whole used to be more aligned with engineering but this has changed considerably since, say, the rise of "Agile". I'd argue that it changed once smaller businesses could afford (and have room for) their own computer. My best guess would be somewhere around the PDP-8 era (late 1960s-early 1970s). Once things started to move away from large-scale computer rooms and consoles of blinking lights, towards a more "hands-on" and interactive approach, where the programmer didn't have to wait between "batches" of runs to see what their code did (correct or not), that is when (from a software development point of view) so-called "engineering" went out the window.
Agathos 7y ago
Many successful interplanetary probes have launched with significant faults that had to be patched or worked around to complete the mission. Luckily the teams are usually able to do that.
ericls 7y ago
How do you known things are correct for NASA and others?
vbtemp 7y ago
Having started my career in spaceflight software engineering, I can 100% confirm this comment is correct. Having now left the field, working on most other software systems and teams tends to basically be "code-til-it-kinda-works" (despite the paeans to "Agile"). It's been amazing how just applying a little bit of the rigor I developed from earlier on in my career from working on embedded flight systems pays incredible dividends later on. That said, one should have no illusions that NASA is this wonderful hub of innovation and a model for organizations to emulate. Not at all. Definitely not. But there is something to learn from a track record of consistent success flying computers deep in space.
falcolas 7y ago
If you have to qualify your statement with hedge words like "most" and "for a very specific definition" then it's hardly "bullshit". Especially with the plethora of stories about fixing bugs in probes mid-mission, crashes due to poorly sanitized inputs, and byzantine failure conditions.
jgeada 7y ago
I call bullshit on you too. Software is impossible to get correct for the vast majority of cases because there is no specification, nobody even knows or agrees what the specification should be or whether there is even such a thing in the first place. Writing the software is as close as we're ever going to discover a fuzzy description of what the system should be, and the vast majority of the times people change their minds about what they really wanted once they get their hands of that iteration of the implementation. Repeat ad nauseum. The notion that most(*) any non-trivial program can be formally specified is a myth that largely exists in educational institutions or research institutions disconnected from the realities of production software engineering, and has no basis in real life. Yes, there are a handful of exceptions for extremely constrained control problems in avionics, and even those turn out to have errors in the specification.
- mtrycz2 7y ago
  Given all of the comments on here, I think I might have not stated the central point clearly enough. It isn't about getting things 100% correct 100% of the time. No engineering does that. What we like to call "software engineering" isn't actually engineering, and more like a craft than anything. Actual engineering in software is possible. I take issue with calling our field software engineering. It's a buzzword.
  - al_ak 7y ago
    Yet formal correctness is about "getting things 100% correct 100% of the time" to the greatest extent possible (you can mathematically prove the "correctness" of a program using formal methods, but it still has to run on actual hardware and you're assuming there are no bugs in your tooling).
hinkley 7y ago
Margaret Hamilton's pedantry on the Apollo scheduler saved the mission, and possibly lives, when a low priority process started flapping. From what I understand nobody forced her to write the code that way but her own conscience. We know that now, but it's an example of a rule that was not written in blood. Many are. Someone had an intuition and nobody died because of it.

veeralpatel979 7y ago

Writing "correct" software is possible IMO, but is it worth it? For most applications, is it worth spending several times more time and resources in order to make it "correct"? I don't think so.

mristin 7y ago
I always wondered where this multiplicative factor of "several times" comes from. In my experience, writing correct software was marginally slower than writing sloppy software as long as most thinking is done with a pen & paper. Would you mind to elaborate a bit more?
- TeMPOraL 7y ago
  I'm guessing: because of cheap labor. Writing correct and/or fast software involves quite a bit of "don't do stupid things" all across the development process. Doing these things right doesn't add much time to development, but one has to first learn how to do these things right. A fresh and inexperienced developer, or the "one year of experience repeated 10 times" person that only worked at "move fast and break things" project isn't going to have this knowledge, but will be cheaper to hire.
- beagle3 7y ago
  Not OP/GP, but I think it's mostly about the definition of correct. Does it correctly handle every possible sequence of inputs? For the vast majority of software in use today, the answer is "no"; The follow up question is "does it matter?" and (luckily or unluckily) the answer for the vast majority of software in the vast majority of use cases, is "no" as well. But occasionally it does, e.g. https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 is a famous poster child, and I quote: >>> The failure occurred only when a particular nonstandard sequence of keystrokes was entered on the VT-100 terminal which controlled the PDP-11 computer: an "X" to (erroneously) select 25 MeV photon mode followed by "cursor up", "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds Software was not obviously correct or incorrect; In fact, it had been acceptable for an earlier model (which had some hardware protections missing from the newer one). Reaching the incorrect state required the race described above to trigger, which did, in fact, happen in practice a handful of times. You can work very hard to formally prove your implementation, only to find out that the compiler had a bug and makes your software bad. Or the CPU does; Or all the designs are fine, but there's an bit flip due to electro-migration or cosmic rays. Many people consider this "force majeur" - "an act of god" one cannot anticipate, but cosmic rays are in fact an expected -- and hard to avoid -- input to many systems. You are in control of a logical model which you can, with extra work, do (provably) correctly. But that IS, from experiencce, 10x to 100x more expensive, and unless you go for 10x-100x more expensive hardware, reduces and moves the sloppiness factor around, but does not eliminate it.
mojuba 7y ago
Depends on what consequences your software can have in the real world. Most of software that you run on the laptop can glitch and crash all it wants as long as your users can tolerate that, but software for cars for example can have very real consequences on lives. We should be worried about poor development practices in the automotive industry specifically. Looks like many manufacturers approach the tasks the same way any other hardware manufacturer does, i.e. software is secondary in their minds. Software for a car is kind of the same as software for your next "smart fridge". Among other things, it can be outsourced for example. Whereas, as Bruce Schneier put it, a modern car is a computer on wheels, not the other way around. A computer on wheels is a lot more dangerous (and there are security implications too, it's what Schneier meant actually).
TeMPOraL 7y ago
It's a complex issue, because often the answer "it's not worth it" is based on company's ability to make someone else pay for the consequences.
ehnto 7y ago
How far down the stack does it need to be correct? Can a 100% correct Python script be considered correct if there could still be bugs further down the stack? Also what's the context of the software? Much software has pretty safe failure modes, but if it's a medical system for example, knowing level of correctness right down to hardware level may be useful. I think in that light, most modern software can be built to be robust and easily maintainable rather than correct. Like the difference between building a bridge, where you understand everything in the system, versus building a vehicle engine, where you have multiple dependencies but you still need it to not catastrophically fail should any of them stop being correct (eg: fuel pump fails, engine management detects lower fuel pressure and cuts ignition, saving the engine).
LandR 7y ago
Everyone seems to think that, but I feel this ends up being death by a thousand cuts. I don't know a single piece of software on my computer right now that doesn't crash, act weird, hang or do something else to annoy me on a daily basis. Now yes, each infraction might be just a minor niggle, but god it adds up! By the end of the day, all the bugs that on their own shouldn't be too bad, leave me frustrated as hell and wanting to say fuck everyone and just go live in the woods. It's especially worse when I know this shit doesn't need to happen.
- LargeWu 7y ago
  Compare that to the cost of software that works perfectly, but costs 10x what it currently does in order to achieve that perfection. Think of all the free (gratis) software you use that maybe would have a huge cost associated with it to be as reliable as you would like. It's all about tradeoffs. Everybody wants cheap software that works perfectly, but reality dictates that can't happen. Most of the time, cheap/free with minor annoyances is better than perfect but high-cost.

bilekas 7y ago

I came to write something simular.. Just because you havent come accross it, doesn't mean its beyond our capabilities, infact, by definition its within our means.

scoutt 7y ago

Yeah, I think the author is being a little pessimistic here, but for the benefit of the doubt it may be interesting to understand what correct means in that sentence. I mean, if correct = 100% uptime, 100% effectiveness, etc., then it's not so much of a software problem, but a hardware one.

mrkeen 7y ago
Yeah but software fails so often and repeatedly that it's rarely worth asking the question "is it software or hardware?"
- scoutt 7y ago
  Exploding batteries, bad wiring, design faults, silicon bugs and erratas, bad connectors, temperature problems... I've seen them all. It's more common than one may usually think, especially if you work in embedded. And somehow I wanted to tie the answer with the concept of the parent about space equipment. BTW, "is it software or hardware?" is exactly what the boss asks first when a customer has a problem.
  - ehnto 7y ago
    Regarding my reply to your parent comment, it sounds like we are both biased by our experience, we might need some data!
    - scoutt 7y ago
      I agreed about the bias. When I read "software" I try to think about a broad spectrum of software applications, including the ones that are running in your fridge, router or the ISS. When author (or most people, I believe) mentions software, it seems it's just apps (like food apps mentioned elsewhere) or web apps, that is just a fraction.
- jonathanstrange 7y ago
  I wouldn't be so sure about that. My PC has an overclocked AMD processor with 32GB non-ECC RAM (because ECC ram wasn't even available when I bought it, let alone affordable). Even if the software was 100% perfect, such a machine is expected to crash once or twice every year or so under full load just from the failure rate of the chips (cosmic radiation / quantum tunnelling effects). Especially RAM seems to have become fairly error-prone nowadays.

onei 7y ago

It's kind of curious coming out of university where I learned about doing requirements properly and up front (which I suspect kind of tends towards a waterfall approach) then comparing that approach to my work now. Project managers constantly change their mind about requirements and then wonder why we're shipping late, although I suspect some of this is not considering failure/bugs when planning. Even when it comes to breaking down and estimating work, if the estimates software engineers produce don't align with the high level schedule they're automatically wrong or people start asking about what we can trim down (which somehow is always testing). Perhaps it's idealism vs reality, but I find it a bit bizarre.

szatkus 7y ago
Because it's much easier to change software than for example a bridge. I don't see it as a bad thing as long as all sides are aware that changing requirements changes the deadline as well.
- reallydontask 7y ago
  > I don't see it as a bad thing as long as all sides are aware that changing requirements changes the deadline as well. In my experience, this could not be further from the truth. There is simply no awareness that change remains expensive it's just that now the process allows it without any fanfare whereas before a change was a big deal. Not saying that change requests are the way to go but certainly an understanding that change is not free would go a long way
  - iainmerrick 7y ago
    Change in a software project definitely isn’t free but I do think it’s fair to say it’s easier than changing a bridge design halfway through. Maybe there’s not as big a difference as people think, though? It occurs to me that is actually possible to change the design of a bridge late in the day. The extra dampers to fix resonance problems on London’s Millennium Bridge are a nice example.
    - 7y ago

jerome-jh 7y ago

It is said that the Apollo 11 lunar lander computer rebooted twice during the final approach. We could probably not say it was correct, but still it was robust. "for any meaningful definition of correct" sounds a bit to strong as an hypothesis. Maybe we could rephrase: for all piece of software S, there exist a definition of correctness for which S is incorrect. Timings in particular are very difficult to ensure.

fjfaase 7y ago
I did not reboot. (I think it did not have the capability to reboot itself.) The program alarms indicated "executive overflows", meaning the guidance computer could not complete all its tasks in real time and had to postpone some of them. https://en.wikipedia.org/wiki/Apollo_11#Lunar_descent https://en.wikipedia.org/wiki/Apollo_11#Lunar_descent See also http://klabs.org/history/apollo_11_alarms/eyles_2004/eyles_2004.htm http://klabs.org/history/apollo_11_alarms/eyles_2004/eyles_2... for a detailed explanation.
- Izkata 7y ago
  > I did not reboot. ...Got something to share with us? ;)

adrianN 7y ago

Writing non-trivial, correct software while not blowing the budget is beyond the capabilities of the human species.

pif 7y ago
Allocating the correct budget to write non-trivial, correct sooftware is beyond the capabilities of idiot managers.
- adrianN 7y ago
  I'm pretty sure almost nobody wants to pay millions of dollars for a copy of Word that never crashes.
  - diafygi 7y ago
    But will they pay millions of dollars for a WiFi firmware stack in energy IoT devices that the grid is increasingly depending on that isn't vulnerable to memory overflows or other hacking vectors? Software is becoming more and more depended on for life and death use cases every day.
    - adrianN 7y ago
      Life-and-death software is already regulated fairly strictly and generally has decent quality. But of course the companies developing it also try to cut costs and in the end it's more about checking off boxes to avoid liability than producing correct software.
      - diafygi 7y ago
        Hmmm, on the sliding scale of harmless to life-and-death software, it seems that as time goes many programs and services migrate from being closer to the harmless end to bring closer to the life-and-death end. I feel that migration is often ignored or discounted. For example, Facebook in the early years was considered mostly harmless, but now has migrated to being exploited by state actors to brainwash populations into hating each other, interfering in elections, or at worst performing genocide on a minority group. We need to stop assuming that just because a software application is harmless now, that it will stay that way, and we need to adjust its "correctness" accordingly as it migrates along the harmless <> life-and-death scale.

yitchelle 7y ago

The effort needed to create a "hello, world" program is not trivial. ie, creation of the tool chain (compiler, linker), the OS which to run the "hello, world executable", the text editor to write the code, the code that is in your keyboard, and the list goes on. All software. No software is trivial if the author wants to broaden his argument to the human species.

tn890 7y ago

imgabe 7y ago
The difference is measuring and quantification. Engineering means you know, or can estimate, a load and perform calculations to see if your design meets that load. Most food order apps seem to be more like, eyeball it, slap something together, then it works until it doesn't, at which point you figure out a fix. That works for a while until something else breaks, so you patch a fix into that. Ad infinitum.
- BigJono 7y ago
  Funny you mention food order apps. UberEats web app is my prime example of how a flaming train wreck can make you a ludicrous amount of money, and how there's almost no correlation between engineering quality (not necessarily product quality) and revenue.
  - exdsq 7y ago
    > there's almost no correlation between engineering quality (not necessarily product quality) and revenue. Agreed. Sadly I'm much more interested in engineering quality than revenue..
  - tatersolid 7y ago
    Uber loses fantastic, mind-bending amounts of money every quarter. Their apps don’t “make money.”

raverbashing 7y ago

There's a very nice graph on "The Mythical Man-Month" about this. Basically, if you constrain your software to work for your specific use case, in your specific system, it's "100x" easier. Also you don't know how many bugs, known or unknown exist in software on space probes.

onion2k 7y ago

most worked correctly, for a very specific definition of correctly. That doesn't mean incorrectness isn't there. That means the circumstances necessary for any incorrectness to manifest in a measurable way didn't happen.

WJW 7y ago
Perhaps, but that is true for any form of engineering. There is no building on earth that can withstand impacts by Texas-sized meteorites, making them incorrectly constructed in those very specific circumstances. Correctness is whether it performs to the requirements of the spec, and if that spec contains tradeoffs that accept non-functioning in certain extreme circumstances then not functioning in those circumstances is NOT incorrect behavior.
- onion2k 7y ago
  I'm not talking about extraordinary events like a giant meteor strike though. I strongly suspect there are plenty of buildings that would collapse if there was a small flood or if they were hit by a car in the wrong place when they have been designed to remain standing in those circumstances. They're incorrectly designed, or incorrectly built, or incorrectly maintained. There's so many ways for things to fail. The only reason they've not collapsed is because those circumstances have never arisen. They probably never will. That doesn't make them correct.

PopeDotNinja 7y ago

Software that doesn't have changing requirements is eventually trivial. You can logic your way to a reasonable solution eventually. Building an app that constantly evolves is never trivial.

exdsq 7y ago
Software without changing requirements is absolutely not eventually trivial, unless ‘eventually’ is a nice word to mask all of the required project complexity. The Mars Rover has specific requirements, is this more trivial than a startups new web-app?

makach 7y ago

I am always amazed when I think about how many instructions a computer goes through every time it does a cold-start and the first pixel is shown on a screen. I particularly liked that you wrote "most worked correctly".

LandR 7y ago

Most software is bad because most developers are bad. I don't think it's more complicated than that.

exdsq 7y ago
Not just that but the discipline is bad because it’s young. The fact that the majority of IT projects fail is just crazy.
zeroxfe 7y ago
Good developers can write bad software too. Primarily because there are a lot more factors to software quality than just developer quality: management quality, team dynamics, cost/time pressure, market dynamics, etc.

lpilot 7y ago

I kind of see where they're coming from with this one, although I don't think it's true. The definitions of trivial and correct can be moved to fit whatever they're trying to say. Rephrasing it as something like it's incredibly difficult to write a large piece of software with no bugs makes more sense. And I think for most organisations it may as well be impossible.

deleted 7y ago

[deleted]

bartread 7y ago

> Thing is, NASA (and Russian and Chinese and Indian and ...) space engineers have approached their challenges from the point of view of actual engineering, while modern "software engineering" is all about craftsmanship, and just a sprinkle of actual engineering to make ourselves feel good. You are, of course, absolutely correct. Context is key here: you treat software engineering like "real" engineering when it's appropriate to do so (space flight, aircraft control systems, nuclear power plant monitoring and control, medical devices, and so forth). However, I think for many organisations - and for much of the time - the engineering aspect is often heavily overplayed, and actually you're trading that off against two key aspects of software that can be used to build a real competitive advantage. Those being (i) speed of implementation, and (ii) malleability. Building software can get you to a solution very quickly. You can also iterate or modify that solution very quickly. Clearly there are limits to this but for many companies, and most startups, getting something working quickly, and iterating on it quickly is far more important than any engineering merit. Same goes for larger companies introducing a new product: don't overegg the "engineering" because it will slow you down and time may be the only sustainable competitive advantage you have. There's a time to be an engineer, and there's a time to be a cowboy hacker (and everything in between). The real trick is in knowing where on the spectrum to execute in the context of the customer base you're serving and the software you're building.

cryptica 7y ago

Even the NASA curiosity rover encountered a software error and now it's stuck. I think OP's argument stands. And NASA software is not that complex if you think about it; at least it has fixed requirements; once they ship it, it's done. No more changes. This is different from some complex software which require constant changes and adaptation to changing requirements; in this case the code has to be designed to adapt easily; this is difficult to achieve and most developers/companies fail at that or in the case of companies, they only just barely succeed through the development of extremely expensive solutions. Big companies like Google don't write efficient software; they use their capital to hire as many expensive engineers as possible to brute force the requirements until it works. That's why they keep rewriting projects from scratch every few years over and over; they don't know how to write adaptable software. Their software is somewhat complex, but still simple enough and they have enough capital so that brute forcing solutions are still feasible (for a high cost). The same cannot be said about more complex tech like blockchain or large-scale machine learning systems; you can't brute force your way to a solution and full rewrites will actually set you back.

exdsq 7y ago
> And NASA software is not that complex if you think about it; at least it has fixed requirements; once they ship it, it's done. No more changes. Nope, they still make changes. This is a case study for a company that helped debug and fix a low-level C bug on the Mars Curiosity Rover as it was traveling to Mars. https://semmle.com/case-studies/semmle-nasa-landing-curiosity-safely-mars https://semmle.com/case-studies/semmle-nasa-landing-curiosit... I work with high-assurance software in the blockchain space and only recently read a paragraph from some old NASA software quality textbook and thought damn.. this is awesome. It's anything but trivial!

arexxbifs 7y ago

I've seen the claim before that it was better in the olden days and that "no real engineer" would create software bugs, especially in connection with space flight. Meanwhile, here is a list of space flight software bugs that has cost over a billion dollars and put human lives in jeopardy: https://en.wikipedia.org/wiki/List_of_software_bugs#Space https://en.wikipedia.org/wiki/List_of_software_bugs#Space

jonathanstrange 7y ago
> I've seen the claim before that it was better in the olden days and that "no real engineer" would create software bugs But who would claim that seriously? That would be obvious hyperbole. The claim is rather that certain teams of engineers have created software/hardware combinations that, for all we know and for all practical purposes, did not have a bug and have not failed due to software error. The development of high integrity systems is costly, though, so the discussion is a bit moot. Sure it would be possible to develop an Android app that - within the limitations of those devices and the buggy operating systems - would not fail due to a bug in its own program. It's just really expensive, particularly if the software is also formally verified. Generally speaking, it doesn't even make sense to consider "high integrity software" without the accompanying hardware. You can only satisfy real-time constraints for specific hardware anyway, and the certification and evaluation should be for software+hardware.
- arexxbifs 7y ago
  I honestly think the post I commented on more or less made this claim by calling the original statement "bullshit", writing off billion-dollar crashes and real risk of death as "programming problems" and calling it "actual engineering". All humans are capable of unkowingly making mistakes which can have dire consequences. We've yet to see an approach to software development that will, without a shadow of a doubt, erradicate the possibility of bugs.

sacado2 7y ago

And it's obviously not the developers' fault, but the clients/customers'. You want a 100% bug-free web browser? OK, so let's define precisely and formally what the browser is supposed to do. Then, you'll sign a very expensive contract, and in 6 months to 1 year, I'll deliver the software, with a formal proof it works as expected. Of course, the requirements won't change during the development process, and any update after delivery will be costly and will take time. Now, who would want to pay thousands of dollars for a web browser that will be delivered in a year and that will be obsolete as soon as a new fancy web tech will be used everywhere else? I can't really blame the customer/client either. But the thing is, "agile" requirements rely on craftmanship, not on engineering. You can only engineer durable things. Buildings would crash constantly, too, if they had to be updated each other week, ASAP, and while keeping the costs as low as possible.

6510 7y ago
What fun proposal! No one wants it but it is exactly what we need. It should do a bunch of really creative things then further new fancy web tech shouldn't exist. Something like: Distribute documents signed by their author in a truly robust way. It should have A review and certification system and It should help monetize publishing in a truly transparent way. Write it in hardware design language and run it in an emulator. :-)
C1sc0cat 7y ago
In the early days of the web 94 I worked on web project for British Telecom. I was told that some one had said well this protocol is all very well but we need to build our own slightly different improved version. Luckily this was stomped on from a great Hight by senior people at BT Labs
TheCoelacanth 7y ago
I think you're underestimating the amount of work involved in creating a web browser by multiple orders of magnitude. If it really took a year to make a bug-free browser, we would have 100 of them instead of 0.
takeda 7y ago
If you want to see extreme cases of this look at code of games. It's no wonder they are so buggy when released.

clarry 7y ago

I think correctness is achievable for a lot of things, but the industry at large isn't chasing it. People ask for a feature. How long does it take, how much does it cost? Dang, can we get it sooner? Nobody ever asked me about what it takes to ensure it is correct. Nobody asked me to work with them to produce a detailed specification that exhaustively considers all cases that must be accounted for, nobody asked for a formal proof, nobody asked me to run a model checker, not even do something like design by contract. And when I look at job postings, all this stuff is completely absent. I was excited when I found one company that recognized Ada and Spark, and got me into an interview for mentioning them. Just sprinkle some assert and make a few tests. If shit stick to ceiling, it's cooked. That's how the industry works. We aren't even trying. It's pretty funny that everyone's making claims about what's possible when nobody is trying. Yep, I bet there are niches, like in aerospace, where some try, and I'm sure some of them also succeed at correctness. We only hear about the failures.

ryandrake 7y ago
Exactly. Correctness (and often mere robustness) is almost never a requirement, so it's never even attempted. There are industries where it is important and where it is done routinely, but evidently not in the places most HN commenters work. To say it's impossible probably sounds kind of silly to the engineers who are currently doing it.
- crimsonalucard 7y ago
  Most HN commenters don't even know that correctness is possible. They don't even know about correctness without testing, they tie correctness and testing together and think it's the same thing. >There are industries where it is important and where it is done routinely, but evidently not in the places most HN commenters work. Can you name these industries?

0x445442 7y ago

Couldn't agree more but I can say it wasn't just NASA. The industry as a whole used to be more aligned with engineering but this has changed considerably since, say, the rise of "Agile".

cr0sh 7y ago
> The industry as a whole used to be more aligned with engineering but this has changed considerably since, say, the rise of "Agile". I'd argue that it changed once smaller businesses could afford (and have room for) their own computer. My best guess would be somewhere around the PDP-8 era (late 1960s-early 1970s). Once things started to move away from large-scale computer rooms and consoles of blinking lights, towards a more "hands-on" and interactive approach, where the programmer didn't have to wait between "batches" of runs to see what their code did (correct or not), that is when (from a software development point of view) so-called "engineering" went out the window.

Agathos 7y ago

Many successful interplanetary probes have launched with significant faults that had to be patched or worked around to complete the mission. Luckily the teams are usually able to do that.

ericls 7y ago

How do you known things are correct for NASA and others?

vbtemp 7y ago

Having started my career in spaceflight software engineering, I can 100% confirm this comment is correct. Having now left the field, working on most other software systems and teams tends to basically be "code-til-it-kinda-works" (despite the paeans to "Agile"). It's been amazing how just applying a little bit of the rigor I developed from earlier on in my career from working on embedded flight systems pays incredible dividends later on. That said, one should have no illusions that NASA is this wonderful hub of innovation and a model for organizations to emulate. Not at all. Definitely not. But there is something to learn from a track record of consistent success flying computers deep in space.

falcolas 7y ago

If you have to qualify your statement with hedge words like "most" and "for a very specific definition" then it's hardly "bullshit". Especially with the plethora of stories about fixing bugs in probes mid-mission, crashes due to poorly sanitized inputs, and byzantine failure conditions.

jgeada 7y ago

I call bullshit on you too. Software is impossible to get correct for the vast majority of cases because there is no specification, nobody even knows or agrees what the specification should be or whether there is even such a thing in the first place. Writing the software is as close as we're ever going to discover a fuzzy description of what the system should be, and the vast majority of the times people change their minds about what they really wanted once they get their hands of that iteration of the implementation. Repeat ad nauseum. The notion that most(*) any non-trivial program can be formally specified is a myth that largely exists in educational institutions or research institutions disconnected from the realities of production software engineering, and has no basis in real life. Yes, there are a handful of exceptions for extremely constrained control problems in avionics, and even those turn out to have errors in the specification.

mtrycz2 7y ago
Given all of the comments on here, I think I might have not stated the central point clearly enough. It isn't about getting things 100% correct 100% of the time. No engineering does that. What we like to call "software engineering" isn't actually engineering, and more like a craft than anything. Actual engineering in software is possible. I take issue with calling our field software engineering. It's a buzzword.
- al_ak 7y ago
  Yet formal correctness is about "getting things 100% correct 100% of the time" to the greatest extent possible (you can mathematically prove the "correctness" of a program using formal methods, but it still has to run on actual hardware and you're assuming there are no bugs in your tooling).

hinkley 7y ago

Margaret Hamilton's pedantry on the Apollo scheduler saved the mission, and possibly lives, when a low priority process started flapping. From what I understand nobody forced her to write the code that way but her own conscience. We know that now, but it's an example of a rule that was not written in blood. Many are. Someone had an intuition and nobody died because of it.