15 ms·

How Is Critical Life or Death Software Tested?

238 points by pablobaz 11y ago

deleted 11y ago
[deleted]
markbnj 11y ago
It's essentially an analog of the concept of tolerance in the physical world of manufacturing and assembly. The less your tolerance for error the more formal and carefully controlled the process, and the more money spent in testing, verification, feedback, and improvement. And yet you can still measure one value in the metric system and another in English units and drill a smoking hole in Mars. It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two?
- pwnna 11y ago
  > It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two? I think you're being overcritical here. It's really really really difficult to reach 100% in any real form of measurement. So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind. I do grant you that this is not specified explicitly in the article, but they do say: "each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."
  - jacquesm 11y ago
    Because it is so difficult you should not make statements that contain fragments such as 'bug free' unless they are preceded by a negation.
  - markbnj 11y ago
    >> So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind. No doubt, and that was in fact my point. I think most of us would be very reluctant to use the phrase "bug free," and in this case his statement was obviously meant to be in stark contrast to that reluctance.
    - pwnna 11y ago
      I think this is more of a technicality that's very subjective. I feel that "no" depends on the probability of something happening as well as how catastrophic it is. For example, after buying a lottery ticket, I can pretty comfortably say that I'm not going to win. Sure, there is a chance, but is it really going to happen? No. As another example, if I put down a waterbottle on my bike seat, there's a 1% chance of the water spilling onto the sidewalk. At this point, I'm also comfortable saying that it's not going to happen. However, if the waterbottle has a 1% chance of exploding when I put it down, I don't think I'll be comfortable saying that it won't happen. The chance of that must be much much lower before I can accept it. I would like to give the person who said that the benefit of the doubt. I'm sure they understand the implication of bug free, but it's likely just easier to say that rather than explaining stats/tradeoffs to a journalist.
- x5n1 11y ago
  Bugs? You mean features!
- eck 11y ago
  > statement about the software being bug free I bet they required the guy who delivered the rocket fuel to sign something saying it contained no impurities, the guy who delivered the external tank to sign something saying it did not leak, etc... why should the software guy be special? Yeah I know. We're special. But the world doesn't always see it that way.
  - superuser2 11y ago
    No. That it contained under n ppm of impurities, that it did not leak more than 0.001%/year, maybe. Only a Sith (and consumer product marketing that is lying to you) deals in absolutes. In the durable goods world, you don't pretend things are perfect. Failure modes are designed and disclosed, replacement of parts is expected and made reasonable, tolerances are marked, failure rate metrics like MTBF are known, and as a customer you choose the price-quality tradeoff that makes sense for you. I just wish consumer products were also sold this way. Instead we pretend every product is awesome and act surprised when things break.
    - tedunangst 11y ago
      At least some consumer goods have similar limits, e.g. maximum amount of arsenic permitted in apple juice. Of course, whenever people find out about these limits, they lose their shit... "Why is any arsenic ok? OMG! The evil government is trying to poison us!"
      - lambdaelite 11y ago
        The person in charge of a mass spectrometer (Thermo Element 2) at a local college likes to toy with his students by running tests on their commercial bottled water during a lab demonstration. This particular instrument can pick up minute traces of uranium, lead, and the like (on the order of fg/L IIRC). At the end of the demonstration, the trash can is full of discarded water bottles.
    - ufmace 11y ago
      That reminded me of phone plans, oddly enough. For a while, everybody was advertising their data plans as unlimited, and then the tech press would get all upset when they found the limit. I always thought the whole thing was dumb - of course there's a limit. Or are we supposed to believe we can push megabytes/s nonstop all month? I'd rather have them just tell me what the limit is and what happens when you go over it than pretend it's unlimited. And stop having the tech press act like the sky is falling when they discover that the unlimited plan actually has a limit.
VLM 11y ago
Could have at least mentioned DO-178C http://en.wikipedia.org/wiki/DO-178C http://en.wikipedia.org/wiki/DO-178C
acomjean 11y ago
I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating the messages when we couldn't get the real equipment in the lab. We were writing software for a radar that didn't yet exist. And it was tested and tested and tested and tested..... The when the pieces came together in integration. They tested and tested. An when they built the physical radar, the software worked. A few configuration things here and there needed adjusting. Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. Was it bug free? Probably not. You can't test every possible scenario. But we took the work seriously. It took longer to get code done than any other environment I've been in, but the code ended up very solid quality.
- outworlder 11y ago
  > Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are performed. At most, there's some sort of metrics and server monitoring, but nothing in the actual software. It's a work of fiction, but Star Trek has these "diagnostics" that are run, with several levels of correctness checks, for pretty much everything. In your typical app, it could be useful to ask it to "run diagnostics", to see if everything was still performing as expected.
  - jerf 11y ago
    It doesn't necessarily take much work to run integration tests as diagnostics on production, especially if you plan on it in advance. I've had good success with it. Of course there's a certain level of destructive testing you can't do live, but that you really ought to do on your development system, load testing being a simple example. It behooves the wise developer to keep these quite separated in the code. :)
x0x0 11y ago
I worked on an emr. Each developer got between 1/2 to 3/2 qa people, in addition to dedicated qa engineers. You had to submit detailed test plans for features. Surprisingly, the company didn't expend much effort on unit tests -- they were there, but not heavily emphasized.
ef4 11y ago
It's all just a question of cost. We know how to write software that comes arbitrary close to perfection. But as defects asymptotically approach zero, cost skyrockets. The interesting question is what technologies can bend that cost/quality curve.
- jerf 11y ago
  This is why this discussion sometimes frustrates me. A lot of the defects we have are because you aren't willing to pay for the sort of software that wouldn't have defects. It's natural to read that as a sort of cynical accusation, but instead, I mean it straight... you really aren't willing to pay what it would take, and you shouldn't be. A $1000 Facebook-access app for your phone (that still somehow has some sort of economies of scale going for it, but that's another discussion) might not crash on you and might take a lot fewer phone resources, but there's no way it's going to be $1000 better than what we get today for free for the vast bulk of users. On the flip side, the cavalier attitude developers who are on the very, very low side of the curve, where huge quality improvements can be obtained for very cheap, towards those cheap practices also frustrates me. What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server? Are you serious? How can you pass up something so cheap, yet so powerful? And I don't just mean, how can you not be bothered to set it up, etc... I mean, as a professional engineer, how can you justify that?
  - manarth 11y ago
    "What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server?" As devil's advocate, why not just run this for me (e.g. on every commit/every push)? Much like the web usability ethos "Why make me think" - why make me work? The lower the barrier to testing - ideally zero, it just happens without the dev having to do anything - the more testing will happen. I don't often get the chance to set things up this way, but when I do, each dev works in their own git branch, and sends a pull-request with their changes. The test server(s) then run the complete test-suite on the branch, and either note the PR with "Tests passed" or emails the dev with "Tests failed" and the reasons. Devs don't need to think about running tests, reviewers/release managers don't need to even consider PRs until the "Tests passed" message shows up…saves time and effort for everyone, and improves code quality. The cost is simply the initial setup time.
jokoon 11y ago
It's true that writing software requires only computers, and that's it would be too expensive to test it in real situations, when the stakes are high maybe it's also important to do live testing ?
- tormeh 11y ago
  Hardware in the loop (HIL) is a sort of middle ground, where all the expensive stuff that breaks are replaced by ordinary computers, but the actual control hardware is not. Corporate video by people who do this, explaining it: https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM
  - neurotech1 11y ago
    The mission systems avionics on a F/A-18F costs around $1-2m per aircraft. This includes Displays, PowerPC based AMC boards, power supplies etc. Hardware in the Loop testing quite practical and was ran as an engineering flight simulator. Missions were flown in the simulator, and problems located without costing $20k/hr per aircraft for an actual flight test. SpaceX do HIL tests with the Falcon 9 & Dragon spacecraft. http://www.spaceflightnow.com/falcon9/003/120424date/ http://www.spaceflightnow.com/falcon9/003/120424date/
mariopt 11y ago
It depends on the software budget and managers. When I worked a company one the CEO's had an idea about adding a camera to road semaphores so that cars wouldn't stop is the road is empty. I asked: What happens if the sun light hits the camera too much? The guy laughed in my face and told I was being ridiculous. I left the company some months later for other reasons but It was pretty scary to me to hear such words at that time. If you're writing software that might cost someone else's life, the budget shouldn't be a limitation nor time. I'm not aware of any software security regulation, If I'm going to release critical software, Does some entity exists to validates my code? This kind of stuff should exist to block companies that only see the profit of getting a contract.
- manarth 11y ago
  Genuinely, in road safety, the company typically considers and weighs the cost of fixing an issue, vs the cost of lawsuits in the event of death/injury. One paper on the topic talks about the Ford Pinto fuel system design: http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Leggett-pinto.html http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Legg... The GM ignition-switch recall also sparked a similar debate: http://en.wikipedia.org/wiki/2014_General_Motors_recall http://en.wikipedia.org/wiki/2014_General_Motors_recall So it's not uncommon that economics outweighs risk-to-life in a lot of businesses.
  - rodgerd 11y ago
    Which is one reason people should be a lot more suspicious of "tort reform" than they are.
istvan__ 11y ago
The article bit misleading, almost suggests that software correctness is achieved by testing, while this is definitely not true. Just for the record, none of the other technical fields using testing for safety critical engineering products as the main solution to ensure safety. This is not too different for software components in mission critical use cases. There are several ways to build a reliable system out of unreliable parts (like combining 3 different units by the 2 out of 3 principle, etc.) but testing is just not the way. It is a nice to have thing.
zyxley 11y ago
For people interested in the idea of formal verification, you may want to look at TLA+ (https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B) and PlusCal (https://en.wikipedia.org/wiki/PlusCal https://en.wikipedia.org/wiki/PlusCal), which have been mentioned on HN before. They're a specialty system for writing code (and mathematical proofs) where every possible system behavior for a given range of inputs can be examined for safety (outputs within allowed ranges with no unexpected behavior) and for liveliness (the expected progression from one output to another).
cesarbs 11y ago
> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and engineers, but of the Process. Serious question: where do I get a job like this? It's my dream way of programming professionally.
- acomjean 11y ago
  Large corporations with government contracts have this kind of work. Probably financial institutions too. The coding proces is slower than most people are used too and can become frustrating.
  - DannoHung 11y ago
    Financial institutions do not necessarily do things this way. Some parts might, but I don't have any experience with them. The parts I do have experience with it is utterly a miracle that anything works.
    - noir_lord 11y ago
      > The parts I do have experience with it is utterly a miracle that anything works. Is my feeling about everything industry I've worked in. The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.
      - aswanson 11y ago
        Seriously? Billing code? I would have imagined that code would be so subject to customer complaint that it would be forced into quality.
        jacobsenscott 11y ago
bargl 11y ago
I'm currently writing software that is non-critical for satellites. It's non-critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back. We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So we are re-writing. That said, accuracy is number one. We have a pretty solid method for testing so far. We know have some robust input scenarios and we know that we want to get a specific output. So we are able to do fairly robust automated "regression" testing. If the numbers don't match then we have an issue, and we have to fix that before moving on. After every validation that the new code gives us acceptable margins of error we wrap it up with unit tests so that we can then modify the code to try to optimize. Our testing is integrated from the highest level of the code down (I know backwards) but that's how we know we can validate the input. We have a lot of testing and a long schedule. If this weren't critical software we'd have a much shorter turn around on what we are writing. We also work very closely with subject matter experts on every change we make. We have a guy who's been working with this software (and the subsequent theory) for 20 years. He's open to change, but he also validates everything so we don't accidentally change the output when we're optimizing.
- david-given 11y ago
  C# is an odd choice for avionics software, surely? It's got a really heavyweight runtime and nondeterministic behaviour due to the garbage collector. I assume this is the satellite's application layer and isn't real time? ...and what are you running it on? I wasn't aware of any embedded operating systems which supported C#!
  - snops 11y ago
    I'm guessing its actually running on a ground station, providing commands or analysing data from satellites, hence why its non-critical and the need has arisen for parallel computation, presumably to speed it up. Many ARM embedded systems can support C# with the open source .net micro framework (http://www.netmf.com http://www.netmf.com), which doesnt require any OS, and was originally developed for Microsoft's SPOT watch. I haven't used it myself, and I agree with you that it doesn't sound, ideal at first for realtime apps, but none or soft realtime embedded applications are common too.
mavdi 11y ago
Not sure if true, but I heard a couple of researchers in my uni wrote some code for Boeing. They wrote the code in Prolog. I assume because it's easier to formal test it. And they didn't trust the compiler, so they had to check the generated machine code line by line. Apparently.
- tjr 11y ago
  If it was DO-178B/C Level A, then yeah, they'd have to inspect the machine code. Regardless of trusting the compiler. And the compiler would have to be inspected too.
amelius 11y ago
If Amazon can do it, then surely an airplane manufacturer can do it [1] [1] http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services-uses-formal-methods/fulltext http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...
- kbenson 11y ago
  As much as that helps, it doesn't yield software, or systems, without faults. The AWS wikipedia page lists a few major outages of AWS[1]. My recollection is that they are generally caused by cascading failures and complex service interrelationships, which are likely much harder to formally prove, if possible. 1: http://en.wikipedia.org/wiki/Amazon_Web_Services#History http://en.wikipedia.org/wiki/Amazon_Web_Services#History
- packetslave 11y ago
  that Amazon paper is talking about design/algorithm verification using formal methods. This does NOT verify that the design or algorithm in question is actually implemented correctly and matches the design perfectly.
  - lambdaelite 11y ago
    And as I recall, TLA+ doesn't tie into code generation which doesn't help with the verification process.
airza 11y ago
My experience with testing avionics controllers was: Everyone seemed to have the correct idea (that bugs were basically Not Allowed.) The company set up enough testing so that those bugs were eventually eliminated. However, the difference between good projects and bad projects was mostly the amount of time and money that this took. My favorite was one where we had an entire test harness written in Python that could completely control the operation of the device being tested in a way that resembled human input. Code was first written in Ada against a monolithic requirements document, while testers wrote their standard test cases against the same document. After the exhaustive amount of testing that took place by developers, testers themselves had the freedom to create contrived test cases that might have escaped the attention of devs (What if we just turn the machine on and off 10 times, because why not?). This had the advantages of a formal software process as well as the ability to exploit human creativity. It also led to me losing a bet that a doppler radar can't be fooled with an empty potato chip bag.
- nchelluri 11y ago
  Care to elaborate on the potato chip bag story? Sounds intriguing.
  - mbrameld 11y ago
    My wild guess loosely based on my time spent writing software to analyze observed radar signals: I believe the parent probably opened the bag along the seams to get one flat piece of reflective (on the inside of the bag) material. I know weather balloons are often reflective so they can be tracked by radar. I'm guessing a full-size potato chip bag opened flat would be big enough for the radar to see, couldn't tell you if that's what the parent meant by fooling the radar or if something else was done to the bag to make the radar interpret it as an aircraft.
    - airza 11y ago
      This is pretty much it. My boss was definitely a level 99 duct tape programmer, and this was one of the many examples of his ingenuity.
guelo 11y ago
The kind of process needed to achieve high quality is just not that fun. Most of the programmers I know would run screaming complaining about the bureaucracy and red tape.
Aleman360 11y ago
Worked on anesthesia machines for a few years. Since both hardware and software is involved, the testing was quite extensive. * Lots of manual testing. While we did unit testing and some automated integration testing, most defects were found using exhaustive manual testing by trained engineers. * Randomized UI testing. Used UI automation to exercise the UI with various physical configurations of the system. Would often run this overnight on many systems and analyze failures every day. * Extensive hazard analysis. Basically, we wrote down everything that could possibly go wrong with the system (including things like gamma radiation), estimated the likelihood and harm, and then listed mitigations. The entire system could run safely even if there was full power failure. "Fail safe" * Detailed software specifications, each of which was linked to manual test cases. Test cases were signed off when executed. * Animal testing for validation. We went to a vet school and put a bunch of dogs under and brushed their teeth. * Limited release for production. We would launch the system at one or two hospitals and monitor it for a few weeks before broader release.
- manarth 11y ago
  You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of hardware interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.
  - Aleman360 11y ago
    Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.
    - david-given 11y ago
      Interlocks are usually fairly crude safety measures, normally in hardware, to make sure that particular combinations of events cannot happen. The Therac-25 is a famous comp.risks cautionary tale. Among the many, many design misfeatures (if you haven't come across it, it's worth a read) was the one that killed people: It was capable of providing two kinds of radiation therapy; electron beam radiation and X-ray radiation. It worked by having an electron beam generator which could be operated at either high power or lower power. Low power was used directly. High power was only used to irradiate a tungsten target which produced X-rays. (I'm simplifying here.) You can probably guess what went wrong; people were directly exposed to the high power electron beam. Several of them died. The obvious interlock here (which apparently previous versions had) was to have a mechanical switch which would only enable the high-power beam when the tungten target was rotated into place. No target, no high power. Simple and relatively foolproof (although it's possible for interlocks to go wrong too).
alkonaut 11y ago
I think in the "normal" software industry we have a skewed picture of quality, simply because it's not the primary focus. For everyday software, it's fine if it works to 95%. Even bugs you have identified must be weighed against new features, and features often win. For the customer it's better to have a piece of software that has all the features they need, but calculates a bad result every 1/1000 times, or crashes a couple of times per day, than a program that doesn't have all the features. It's also the "release early release often" thing where the cheapest testers are your end users. That is far from the "release once patch never" of rockets. So feature bloat and suffering quality isn't really due to bad practices, it's an active choice.
danieltillett 11y ago
Have languages been written that are designed to be maximally orthangonal to each other in regards introducing bugs? If so what combinations are used?
rexignis 11y ago
Additional reading material: some of NASA's own rules for safe code (with explanations of each! love the stack one, free memory management :P). http://spinroot.com/gerard/pdf/P10.pdf http://spinroot.com/gerard/pdf/P10.pdf
- rzzzt 11y ago
  JPL also published a C coding standard, which details language constructs that one should and shouldn't use in a mission critical embedded system. Some of the rules make a reappearance there (the "Power of Ten" article is mentioned in the introduction). http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
coderjames 11y ago
I think the article might be mistaken about one point: "For one thing, the Boeing approach is going out of style or has mostly gone out of style, according to SE poster Uri Dekel (handle: Uri), a Google software engineer." It absolutely has not gone out of style in avionics software engineering. As a person who writes software for avionics, I can say that extensive design reviews at every step combined with rigorous testing is exactly how we build software. That's how its done at every avionics software company I've ever worked at (3 so far). Formal methods are generally still too cutting-edge and complicated for many people in this industry. So maybe Google doesn't bother with design reviews, but those of us writing life-or-death software definitely do.
- SnacksOnAPlane 11y ago
  He meant the practice of sending the software engineers up on the first flight.
lambdaelite 11y ago
Testing is certainly very important, but I think it should be emphasized that testing is but one part of the software development life cycle that produces an acceptably safe product.
deleted 11y ago
[deleted]
- manarth 11y ago
  "ship both a and b, whose output always has to match exactly or it reports a fault and the component requires replacement." And when that software disagrees…but the plane happens to be at 40,000 feet? The plane just stops running until the component is replaced? I don't know too much about plane hardware, but I scuba-dive a rebreather which has critical life-support electronics. It has two independent computers, and three O2 oxygen-pressure cells. The 3 cells report their reading of O2 pressure to both computers. Both computers simultaneously display the pressure on independent displays. One computer is primary (active, controlling the O2 pressure), whilst the secondary is display-only. Both computers use a majority-rule…the two oxygen cells with the closest value win, whilst the third is ignored. This could potentially be fatal - two failing cells can report incorrect pressures and win the vote - so rebreather divers are also taught manual techniques to validate the computer readings (such as a diluent flush which is expected to produce a known predictable reading). So there are a few techniques beyond simple a/b testing: best of 3 (or more, if available); independent circuits (ideally designed + built by independent manufacturers); manual techniques to give human verification of the data. Whilst I certainly don't have extensive knowledge of real-time/safety-critical systems, it's clear that there are a lot of techniques, processes and procedures that we wouldn't necessarily be aware of in unrelated tech (e.g. web-dev) that do directly relate to that subject, and might well solve many of the scenarios we come up with.
  - deleted 11y ago
    [deleted]
  - mitchty 11y ago
    I have a friend that used to do embedded programming for nuclear power plants. From what he said they have 5 systems, two running software from vendor a, two from vendor b, and another running software from vendor c for failsafe. A quorum have to agree, if any don't it gets pulled out of service and analyzed why it wasn't the same output. If you get down to one there is still a mechanical failover but at that point you're already bringing things down. 3 at any time was considered a critical failure. Two at once would be full stop time. I'd presume planes are similar. I know the space shuttle did the same at least so I assume the technique is common.
Davesjoshin 11y ago
As a professional tester, I find this pretty interesting. Always wondered how these ultra critical programs were tested. (I work in the creative website space)
jarrettch 11y ago
I've been wondering about this lately. My stepdad has heart failure and they put a heart pump in his chest. It regulates blood flow and settings can be changed, etc. Leave it to a dev to think "How much testing has gone into this thing?" Even one minor slip-up in his blood flow, either too high or too low, could mean a stroke and possibly death. Or, God forbid, the thing crashes somehow and stops working. That machine is much easier to code than a space shuttle obviously, but I still wondered about it. The tech has to be rock solid. Even one malfunction could cause so much despair in a family, and could also cost your company millions.
- mrsteveman1 11y ago
  I don't know much about heart pumps, but I do have an implanted pacer/defibrillator, and I'm currently not too happy with the people who made it. I've been told by my cardiologist (and engineers working for the manufacturer) that it doesn't "fail safe" if the battery level drops too low to keep the device running (which is inevitable if it isn't replaced after 7-8 years, but it can and sometimes does happen prematurely and without warning). In that situation, not only can it suddenly become unable to correct an arrhythmia (as expected), it could actually cause one all by itself, or pace above 200BPM for no reason. No one I've talked to in the healthcare industry seems at all surprised about this for some reason. They just started monitoring it more often the closer it got to the "replace me now" indicator level.
brudgers 11y ago
No discussion of mission critical software is complete without mentioning Margaret Hamilton: https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist...
zobzu 11y ago
"There's a serious move towards formal verification rather than random functional testing," he writes. "Government agencies like NASA and some defense organizations are spending more and more on these technologies. They're still a PITA [pain in the ass] for the average programmer, but they're often more effective at testing critical systems." this part is kinda cool because there's also always been a movement for toward this for opensource/commercial security software, at least for the kernel. Of course, we have no such thing today (as in i doubt anyone here runs such a system in production), but the interest is there.
anoopelias 11y ago
I always thought that the way to ensure zero failures at a level is to ensure accuracy at the next level. For example, if all trains runs with a minute accuracy at every points, then there won't be any collision by mistake.
imrehg 11y ago
Okay, the LightSail spacecraft is not a "life or death" thing, but, as it was in the news yesterday[1] seriously logging into CSV file onboard can crash the system and have to wait until it reboots itself (don't react to soft reboot either)? Feels sad, that a lot of lessons learned are getting lost along the way... [1]: http://www.planetary.org/blogs/jason-davis/2015/20150526-software-glitch-pauses-ls-test.html http://www.planetary.org/blogs/jason-davis/2015/20150526-sof...
henrik_w 11y ago
The story of the Boeing engineers flying on the test flights is a perfect example of "skin in the game" (from Antifragile by Nassim Nicholas Taleb). Here's what I wrote about that in a blog post on Antifragility and SW development: At the end of the book, there is a chapter on ethics that Taleb calls “skin in the game”. To have skin in the game, you should share both in the upside and downside. Taleb quotes the 3,800 year old Hammurabi’s code: “If a builder builds a house and the house collapses and causes the death of the owner – the builder shall be put to death”. It is interesting to view this from a software development perspective. I have never worked on software where people’s lives were in danger if the software failed, but I would not be willing to submit to Hammurabi’s code if I did. But I think a little less extreme form of skin in the game is actually very good. Being on call for example. If the software you wrote fails, you may get called in the middle of the night to help fix it. I have been on call at most of the places I have worked in the past, and I think it has a lot of benefits. It gives you an incentive to be very thorough, both in development and testing. It also forces you to make the software debuggable – otherwise you yourself will suffer. Another way of introducing skin in the game is dog-fooding – using the software you are developing in your daily work. I have never worked on software that we have been able to dog-food, but I think that is another great practice. http://henrikwarne.com/2014/06/08/antifragility-and-software-development/ http://henrikwarne.com/2014/06/08/antifragility-and-software...
- dbdr 11y ago
  The examples you are giving are all about sharing the downside. What are the options for the upside?
crucini 11y ago
I think the right answer here is to develop languages or libraries that enforce the constraints you want. I think we should beware of adding layers of "silver bullet" technologies that promise to fix the last crisis, but increase the friction of development. To illustrate what I mean, PLCs are normally programmed in ladder logic. From that level, I don't think you can crash the machine or corrupt memory. So the risks are limited to the "operating system" if you will, which can be more mature and tested than the "application".

238 points by pablobaz 11y ago

deleted 11y ago

[deleted]

It's essentially an analog of the concept of tolerance in the physical world of manufacturing and assembly. The less your tolerance for error the more formal and carefully controlled the process, and the more money spent in testing, verification, feedback, and improvement. And yet you can still measure one value in the metric system and another in English units and drill a smoking hole in Mars. It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two?

pwnna 11y ago
> It was sort of striking to read Charles Fishman's statement about the software being bug free, followed immediately by the supporting fact that the last three versions had one bug each. If they had one bug, how are you 100% sure they didn't have two? I think you're being overcritical here. It's really really really difficult to reach 100% in any real form of measurement. So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind. I do grant you that this is not specified explicitly in the article, but they do say: "each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."
- jacquesm 11y ago
  Because it is so difficult you should not make statements that contain fragments such as 'bug free' unless they are preceded by a negation.
- markbnj 11y ago
  >> So I think when they mean "bug free", they probably mean the chance of a bug is below some threshold of probability. The famous six sigma rule comes in mind. No doubt, and that was in fact my point. I think most of us would be very reluctant to use the phrase "bug free," and in this case his statement was obviously meant to be in stark contrast to that reluctance.
  - pwnna 11y ago
    I think this is more of a technicality that's very subjective. I feel that "no" depends on the probability of something happening as well as how catastrophic it is. For example, after buying a lottery ticket, I can pretty comfortably say that I'm not going to win. Sure, there is a chance, but is it really going to happen? No. As another example, if I put down a waterbottle on my bike seat, there's a 1% chance of the water spilling onto the sidewalk. At this point, I'm also comfortable saying that it's not going to happen. However, if the waterbottle has a 1% chance of exploding when I put it down, I don't think I'll be comfortable saying that it won't happen. The chance of that must be much much lower before I can accept it. I would like to give the person who said that the benefit of the doubt. I'm sure they understand the implication of bug free, but it's likely just easier to say that rather than explaining stats/tradeoffs to a journalist.
x5n1 11y ago
Bugs? You mean features!
eck 11y ago
> statement about the software being bug free I bet they required the guy who delivered the rocket fuel to sign something saying it contained no impurities, the guy who delivered the external tank to sign something saying it did not leak, etc... why should the software guy be special? Yeah I know. We're special. But the world doesn't always see it that way.
- superuser2 11y ago
  No. That it contained under n ppm of impurities, that it did not leak more than 0.001%/year, maybe. Only a Sith (and consumer product marketing that is lying to you) deals in absolutes. In the durable goods world, you don't pretend things are perfect. Failure modes are designed and disclosed, replacement of parts is expected and made reasonable, tolerances are marked, failure rate metrics like MTBF are known, and as a customer you choose the price-quality tradeoff that makes sense for you. I just wish consumer products were also sold this way. Instead we pretend every product is awesome and act surprised when things break.
  - tedunangst 11y ago
    At least some consumer goods have similar limits, e.g. maximum amount of arsenic permitted in apple juice. Of course, whenever people find out about these limits, they lose their shit... "Why is any arsenic ok? OMG! The evil government is trying to poison us!"
    - lambdaelite 11y ago
      The person in charge of a mass spectrometer (Thermo Element 2) at a local college likes to toy with his students by running tests on their commercial bottled water during a lab demonstration. This particular instrument can pick up minute traces of uranium, lead, and the like (on the order of fg/L IIRC). At the end of the demonstration, the trash can is full of discarded water bottles.
  - ufmace 11y ago
    That reminded me of phone plans, oddly enough. For a while, everybody was advertising their data plans as unlimited, and then the tech press would get all upset when they found the limit. I always thought the whole thing was dumb - of course there's a limit. Or are we supposed to believe we can push megabytes/s nonstop all month? I'd rather have them just tell me what the limit is and what happens when you go over it than pretend it's unlimited. And stop having the tech press act like the sky is falling when they discover that the unlimited plan actually has a limit.

VLM 11y ago

Could have at least mentioned DO-178C http://en.wikipedia.org/wiki/DO-178C http://en.wikipedia.org/wiki/DO-178C

acomjean 11y ago

I wrote software for Radars. Kind of important (not like plane software). We used Ada alot, which in my estimation helped. Software was reviewed. Tests were reviewed. Reliability was favored over other things (for example recursion was discouraged). We used Ada's constrained types (this value is between 1 and 99, if it goes out of range, throw an exception). For external hardware inputs, we had software simulating the messages when we couldn't get the real equipment in the lab. We were writing software for a radar that didn't yet exist. And it was tested and tested and tested and tested..... The when the pieces came together in integration. They tested and tested. An when they built the physical radar, the software worked. A few configuration things here and there needed adjusting. Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. Was it bug free? Probably not. You can't test every possible scenario. But we took the work seriously. It took longer to get code done than any other environment I've been in, but the code ended up very solid quality.

outworlder 11y ago
> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running. I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are performed. At most, there's some sort of metrics and server monitoring, but nothing in the actual software. It's a work of fiction, but Star Trek has these "diagnostics" that are run, with several levels of correctness checks, for pretty much everything. In your typical app, it could be useful to ask it to "run diagnostics", to see if everything was still performing as expected.
- jerf 11y ago
  It doesn't necessarily take much work to run integration tests as diagnostics on production, especially if you plan on it in advance. I've had good success with it. Of course there's a certain level of destructive testing you can't do live, but that you really ought to do on your development system, load testing being a simple example. It behooves the wise developer to keep these quite separated in the code. :)

x0x0 11y ago

I worked on an emr. Each developer got between 1/2 to 3/2 qa people, in addition to dedicated qa engineers. You had to submit detailed test plans for features. Surprisingly, the company didn't expend much effort on unit tests -- they were there, but not heavily emphasized.

ef4 11y ago

It's all just a question of cost. We know how to write software that comes arbitrary close to perfection. But as defects asymptotically approach zero, cost skyrockets. The interesting question is what technologies can bend that cost/quality curve.

jerf 11y ago
This is why this discussion sometimes frustrates me. A lot of the defects we have are because you aren't willing to pay for the sort of software that wouldn't have defects. It's natural to read that as a sort of cynical accusation, but instead, I mean it straight... you really aren't willing to pay what it would take, and you shouldn't be. A $1000 Facebook-access app for your phone (that still somehow has some sort of economies of scale going for it, but that's another discussion) might not crash on you and might take a lot fewer phone resources, but there's no way it's going to be $1000 better than what we get today for free for the vast bulk of users. On the flip side, the cavalier attitude developers who are on the very, very low side of the curve, where huge quality improvements can be obtained for very cheap, towards those cheap practices also frustrates me. What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server? Are you serious? How can you pass up something so cheap, yet so powerful? And I don't just mean, how can you not be bothered to set it up, etc... I mean, as a professional engineer, how can you justify that?
- manarth 11y ago
  "What do you mean you can't be bothered to run the automated test suite that I have handed you on a silver platter on your build server?" As devil's advocate, why not just run this for me (e.g. on every commit/every push)? Much like the web usability ethos "Why make me think" - why make me work? The lower the barrier to testing - ideally zero, it just happens without the dev having to do anything - the more testing will happen. I don't often get the chance to set things up this way, but when I do, each dev works in their own git branch, and sends a pull-request with their changes. The test server(s) then run the complete test-suite on the branch, and either note the PR with "Tests passed" or emails the dev with "Tests failed" and the reasons. Devs don't need to think about running tests, reviewers/release managers don't need to even consider PRs until the "Tests passed" message shows up…saves time and effort for everyone, and improves code quality. The cost is simply the initial setup time.

jokoon 11y ago

It's true that writing software requires only computers, and that's it would be too expensive to test it in real situations, when the stakes are high maybe it's also important to do live testing ?

tormeh 11y ago
Hardware in the loop (HIL) is a sort of middle ground, where all the expensive stuff that breaks are replaced by ordinary computers, but the actual control hardware is not. Corporate video by people who do this, explaining it: https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM https://www.youtube.com/watch?t=116&v=YpxPAuHNpdM
- neurotech1 11y ago
  The mission systems avionics on a F/A-18F costs around $1-2m per aircraft. This includes Displays, PowerPC based AMC boards, power supplies etc. Hardware in the Loop testing quite practical and was ran as an engineering flight simulator. Missions were flown in the simulator, and problems located without costing $20k/hr per aircraft for an actual flight test. SpaceX do HIL tests with the Falcon 9 & Dragon spacecraft. http://www.spaceflightnow.com/falcon9/003/120424date/ http://www.spaceflightnow.com/falcon9/003/120424date/

mariopt 11y ago

It depends on the software budget and managers. When I worked a company one the CEO's had an idea about adding a camera to road semaphores so that cars wouldn't stop is the road is empty. I asked: What happens if the sun light hits the camera too much? The guy laughed in my face and told I was being ridiculous. I left the company some months later for other reasons but It was pretty scary to me to hear such words at that time. If you're writing software that might cost someone else's life, the budget shouldn't be a limitation nor time. I'm not aware of any software security regulation, If I'm going to release critical software, Does some entity exists to validates my code? This kind of stuff should exist to block companies that only see the profit of getting a contract.

manarth 11y ago
Genuinely, in road safety, the company typically considers and weighs the cost of fixing an issue, vs the cost of lawsuits in the event of death/injury. One paper on the topic talks about the Ford Pinto fuel system design: http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Leggett-pinto.html http://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Legg... The GM ignition-switch recall also sparked a similar debate: http://en.wikipedia.org/wiki/2014_General_Motors_recall http://en.wikipedia.org/wiki/2014_General_Motors_recall So it's not uncommon that economics outweighs risk-to-life in a lot of businesses.
- rodgerd 11y ago
  Which is one reason people should be a lot more suspicious of "tort reform" than they are.

istvan__ 11y ago

The article bit misleading, almost suggests that software correctness is achieved by testing, while this is definitely not true. Just for the record, none of the other technical fields using testing for safety critical engineering products as the main solution to ensure safety. This is not too different for software components in mission critical use cases. There are several ways to build a reliable system out of unreliable parts (like combining 3 different units by the 2 out of 3 principle, etc.) but testing is just not the way. It is a nice to have thing.

zyxley 11y ago

For people interested in the idea of formal verification, you may want to look at TLA+ (https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B) and PlusCal (https://en.wikipedia.org/wiki/PlusCal https://en.wikipedia.org/wiki/PlusCal), which have been mentioned on HN before. They're a specialty system for writing code (and mathematical proofs) where every possible system behavior for a given range of inputs can be examined for safety (outputs within allowed ranges with no unexpected behavior) and for liveliness (the expected progression from one output to another).

cesarbs 11y ago

> As for the code itself, its perfection came as the result of basically the opposite of every trope normally assigned to "coder." Creativity in the shuttle group was discouraged; shifts were nine-to-five; code hotshots and superstars were not tolerated; over half of the team consisted of women; debugging barely existed because mistakes were the rarest of occurrences. Programming was the product not of coders and engineers, but of the Process. Serious question: where do I get a job like this? It's my dream way of programming professionally.

acomjean 11y ago
Large corporations with government contracts have this kind of work. Probably financial institutions too. The coding proces is slower than most people are used too and can become frustrating.
- DannoHung 11y ago
  Financial institutions do not necessarily do things this way. Some parts might, but I don't have any experience with them. The parts I do have experience with it is utterly a miracle that anything works.
  - noir_lord 11y ago
    > The parts I do have experience with it is utterly a miracle that anything works. Is my feeling about everything industry I've worked in. The stuff that runs telecoms (mostly billing side) particularly is the stuff of nightmares.
    - aswanson 11y ago
      Seriously? Billing code? I would have imagined that code would be so subject to customer complaint that it would be forced into quality.
      - jacobsenscott 11y ago

bargl 11y ago

I'm currently writing software that is non-critical for satellites. It's non-critical in the manner that if we get things wrong our company will lose millions of dollars but the satellite won't burn any resources that it can't get back. We are currently porting code over from C++ to a C# system with parallel computation. The current system has been flying for a long time but has no testing and is tied to a bad UI. So we are re-writing. That said, accuracy is number one. We have a pretty solid method for testing so far. We know have some robust input scenarios and we know that we want to get a specific output. So we are able to do fairly robust automated "regression" testing. If the numbers don't match then we have an issue, and we have to fix that before moving on. After every validation that the new code gives us acceptable margins of error we wrap it up with unit tests so that we can then modify the code to try to optimize. Our testing is integrated from the highest level of the code down (I know backwards) but that's how we know we can validate the input. We have a lot of testing and a long schedule. If this weren't critical software we'd have a much shorter turn around on what we are writing. We also work very closely with subject matter experts on every change we make. We have a guy who's been working with this software (and the subsequent theory) for 20 years. He's open to change, but he also validates everything so we don't accidentally change the output when we're optimizing.

david-given 11y ago
C# is an odd choice for avionics software, surely? It's got a really heavyweight runtime and nondeterministic behaviour due to the garbage collector. I assume this is the satellite's application layer and isn't real time? ...and what are you running it on? I wasn't aware of any embedded operating systems which supported C#!
- snops 11y ago
  I'm guessing its actually running on a ground station, providing commands or analysing data from satellites, hence why its non-critical and the need has arisen for parallel computation, presumably to speed it up. Many ARM embedded systems can support C# with the open source .net micro framework (http://www.netmf.com http://www.netmf.com), which doesnt require any OS, and was originally developed for Microsoft's SPOT watch. I haven't used it myself, and I agree with you that it doesn't sound, ideal at first for realtime apps, but none or soft realtime embedded applications are common too.

mavdi 11y ago

Not sure if true, but I heard a couple of researchers in my uni wrote some code for Boeing. They wrote the code in Prolog. I assume because it's easier to formal test it. And they didn't trust the compiler, so they had to check the generated machine code line by line. Apparently.

tjr 11y ago
If it was DO-178B/C Level A, then yeah, they'd have to inspect the machine code. Regardless of trusting the compiler. And the compiler would have to be inspected too.

amelius 11y ago

If Amazon can do it, then surely an airplane manufacturer can do it [1] [1] http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services-uses-formal-methods/fulltext http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...

kbenson 11y ago
As much as that helps, it doesn't yield software, or systems, without faults. The AWS wikipedia page lists a few major outages of AWS[1]. My recollection is that they are generally caused by cascading failures and complex service interrelationships, which are likely much harder to formally prove, if possible. 1: http://en.wikipedia.org/wiki/Amazon_Web_Services#History http://en.wikipedia.org/wiki/Amazon_Web_Services#History
packetslave 11y ago
that Amazon paper is talking about design/algorithm verification using formal methods. This does NOT verify that the design or algorithm in question is actually implemented correctly and matches the design perfectly.
- lambdaelite 11y ago
  And as I recall, TLA+ doesn't tie into code generation which doesn't help with the verification process.

airza 11y ago

My experience with testing avionics controllers was: Everyone seemed to have the correct idea (that bugs were basically Not Allowed.) The company set up enough testing so that those bugs were eventually eliminated. However, the difference between good projects and bad projects was mostly the amount of time and money that this took. My favorite was one where we had an entire test harness written in Python that could completely control the operation of the device being tested in a way that resembled human input. Code was first written in Ada against a monolithic requirements document, while testers wrote their standard test cases against the same document. After the exhaustive amount of testing that took place by developers, testers themselves had the freedom to create contrived test cases that might have escaped the attention of devs (What if we just turn the machine on and off 10 times, because why not?). This had the advantages of a formal software process as well as the ability to exploit human creativity. It also led to me losing a bet that a doppler radar can't be fooled with an empty potato chip bag.

nchelluri 11y ago
Care to elaborate on the potato chip bag story? Sounds intriguing.
- mbrameld 11y ago
  My wild guess loosely based on my time spent writing software to analyze observed radar signals: I believe the parent probably opened the bag along the seams to get one flat piece of reflective (on the inside of the bag) material. I know weather balloons are often reflective so they can be tracked by radar. I'm guessing a full-size potato chip bag opened flat would be big enough for the radar to see, couldn't tell you if that's what the parent meant by fooling the radar or if something else was done to the bag to make the radar interpret it as an aircraft.
  - airza 11y ago
    This is pretty much it. My boss was definitely a level 99 duct tape programmer, and this was one of the many examples of his ingenuity.

guelo 11y ago

The kind of process needed to achieve high quality is just not that fun. Most of the programmers I know would run screaming complaining about the bureaucracy and red tape.

Aleman360 11y ago

Worked on anesthesia machines for a few years. Since both hardware and software is involved, the testing was quite extensive. * Lots of manual testing. While we did unit testing and some automated integration testing, most defects were found using exhaustive manual testing by trained engineers. * Randomized UI testing. Used UI automation to exercise the UI with various physical configurations of the system. Would often run this overnight on many systems and analyze failures every day. * Extensive hazard analysis. Basically, we wrote down everything that could possibly go wrong with the system (including things like gamma radiation), estimated the likelihood and harm, and then listed mitigations. The entire system could run safely even if there was full power failure. "Fail safe" * Detailed software specifications, each of which was linked to manual test cases. Test cases were signed off when executed. * Animal testing for validation. We went to a vet school and put a bunch of dogs under and brushed their teeth. * Limited release for production. We would launch the system at one or two hospitals and monitor it for a few weeks before broader release.

manarth 11y ago
You just have to look at Therac 25 for the risks of relying on software interlocks alone. One of the prevailing pieces of feedback was the lack of hardware interlocks - whether that's possible on anesthesia machines I don't know…but the prevailing wisdom is to use/include hardware interlocks wherever that's feasible, for any critical life-supporting equipment.
- Aleman360 11y ago
  Not sure what you mean by "interlocks", but the hardware was quite distributed. Each critical component had its own board and industrial microcontroller. And we had various levels of watchdogs keeping track of system health at all times.
  - david-given 11y ago
    Interlocks are usually fairly crude safety measures, normally in hardware, to make sure that particular combinations of events cannot happen. The Therac-25 is a famous comp.risks cautionary tale. Among the many, many design misfeatures (if you haven't come across it, it's worth a read) was the one that killed people: It was capable of providing two kinds of radiation therapy; electron beam radiation and X-ray radiation. It worked by having an electron beam generator which could be operated at either high power or lower power. Low power was used directly. High power was only used to irradiate a tungsten target which produced X-rays. (I'm simplifying here.) You can probably guess what went wrong; people were directly exposed to the high power electron beam. Several of them died. The obvious interlock here (which apparently previous versions had) was to have a mechanical switch which would only enable the high-power beam when the tungten target was rotated into place. No target, no high power. Simple and relatively foolproof (although it's possible for interlocks to go wrong too).

alkonaut 11y ago

I think in the "normal" software industry we have a skewed picture of quality, simply because it's not the primary focus. For everyday software, it's fine if it works to 95%. Even bugs you have identified must be weighed against new features, and features often win. For the customer it's better to have a piece of software that has all the features they need, but calculates a bad result every 1/1000 times, or crashes a couple of times per day, than a program that doesn't have all the features. It's also the "release early release often" thing where the cheapest testers are your end users. That is far from the "release once patch never" of rockets. So feature bloat and suffering quality isn't really due to bad practices, it's an active choice.

danieltillett 11y ago

Have languages been written that are designed to be maximally orthangonal to each other in regards introducing bugs? If so what combinations are used?

rexignis 11y ago

Additional reading material: some of NASA's own rules for safe code (with explanations of each! love the stack one, free memory management :P). http://spinroot.com/gerard/pdf/P10.pdf http://spinroot.com/gerard/pdf/P10.pdf

rzzzt 11y ago
JPL also published a C coding standard, which details language constructs that one should and shouldn't use in a mission critical embedded system. Some of the rules make a reappearance there (the "Power of Ten" article is mentioned in the introduction). http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf

coderjames 11y ago

I think the article might be mistaken about one point: "For one thing, the Boeing approach is going out of style or has mostly gone out of style, according to SE poster Uri Dekel (handle: Uri), a Google software engineer." It absolutely has not gone out of style in avionics software engineering. As a person who writes software for avionics, I can say that extensive design reviews at every step combined with rigorous testing is exactly how we build software. That's how its done at every avionics software company I've ever worked at (3 so far). Formal methods are generally still too cutting-edge and complicated for many people in this industry. So maybe Google doesn't bother with design reviews, but those of us writing life-or-death software definitely do.

SnacksOnAPlane 11y ago
He meant the practice of sending the software engineers up on the first flight.

lambdaelite 11y ago

Testing is certainly very important, but I think it should be emphasized that testing is but one part of the software development life cycle that produces an acceptably safe product.

deleted 11y ago

[deleted]

manarth 11y ago
"ship both a and b, whose output always has to match exactly or it reports a fault and the component requires replacement." And when that software disagrees…but the plane happens to be at 40,000 feet? The plane just stops running until the component is replaced? I don't know too much about plane hardware, but I scuba-dive a rebreather which has critical life-support electronics. It has two independent computers, and three O2 oxygen-pressure cells. The 3 cells report their reading of O2 pressure to both computers. Both computers simultaneously display the pressure on independent displays. One computer is primary (active, controlling the O2 pressure), whilst the secondary is display-only. Both computers use a majority-rule…the two oxygen cells with the closest value win, whilst the third is ignored. This could potentially be fatal - two failing cells can report incorrect pressures and win the vote - so rebreather divers are also taught manual techniques to validate the computer readings (such as a diluent flush which is expected to produce a known predictable reading). So there are a few techniques beyond simple a/b testing: best of 3 (or more, if available); independent circuits (ideally designed + built by independent manufacturers); manual techniques to give human verification of the data. Whilst I certainly don't have extensive knowledge of real-time/safety-critical systems, it's clear that there are a lot of techniques, processes and procedures that we wouldn't necessarily be aware of in unrelated tech (e.g. web-dev) that do directly relate to that subject, and might well solve many of the scenarios we come up with.
- deleted 11y ago
  [deleted]
- mitchty 11y ago
  I have a friend that used to do embedded programming for nuclear power plants. From what he said they have 5 systems, two running software from vendor a, two from vendor b, and another running software from vendor c for failsafe. A quorum have to agree, if any don't it gets pulled out of service and analyzed why it wasn't the same output. If you get down to one there is still a mechanical failover but at that point you're already bringing things down. 3 at any time was considered a critical failure. Two at once would be full stop time. I'd presume planes are similar. I know the space shuttle did the same at least so I assume the technique is common.

Davesjoshin 11y ago

As a professional tester, I find this pretty interesting. Always wondered how these ultra critical programs were tested. (I work in the creative website space)

jarrettch 11y ago

I've been wondering about this lately. My stepdad has heart failure and they put a heart pump in his chest. It regulates blood flow and settings can be changed, etc. Leave it to a dev to think "How much testing has gone into this thing?" Even one minor slip-up in his blood flow, either too high or too low, could mean a stroke and possibly death. Or, God forbid, the thing crashes somehow and stops working. That machine is much easier to code than a space shuttle obviously, but I still wondered about it. The tech has to be rock solid. Even one malfunction could cause so much despair in a family, and could also cost your company millions.

mrsteveman1 11y ago
I don't know much about heart pumps, but I do have an implanted pacer/defibrillator, and I'm currently not too happy with the people who made it. I've been told by my cardiologist (and engineers working for the manufacturer) that it doesn't "fail safe" if the battery level drops too low to keep the device running (which is inevitable if it isn't replaced after 7-8 years, but it can and sometimes does happen prematurely and without warning). In that situation, not only can it suddenly become unable to correct an arrhythmia (as expected), it could actually cause one all by itself, or pace above 200BPM for no reason. No one I've talked to in the healthcare industry seems at all surprised about this for some reason. They just started monitoring it more often the closer it got to the "replace me now" indicator level.

brudgers 11y ago

No discussion of mission critical software is complete without mentioning Margaret Hamilton: https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist...

zobzu 11y ago

"There's a serious move towards formal verification rather than random functional testing," he writes. "Government agencies like NASA and some defense organizations are spending more and more on these technologies. They're still a PITA [pain in the ass] for the average programmer, but they're often more effective at testing critical systems." this part is kinda cool because there's also always been a movement for toward this for opensource/commercial security software, at least for the kernel. Of course, we have no such thing today (as in i doubt anyone here runs such a system in production), but the interest is there.

anoopelias 11y ago

I always thought that the way to ensure zero failures at a level is to ensure accuracy at the next level. For example, if all trains runs with a minute accuracy at every points, then there won't be any collision by mistake.

imrehg 11y ago

Okay, the LightSail spacecraft is not a "life or death" thing, but, as it was in the news yesterday[1] seriously logging into CSV file onboard can crash the system and have to wait until it reboots itself (don't react to soft reboot either)? Feels sad, that a lot of lessons learned are getting lost along the way... [1]: http://www.planetary.org/blogs/jason-davis/2015/20150526-software-glitch-pauses-ls-test.html http://www.planetary.org/blogs/jason-davis/2015/20150526-sof...

henrik_w 11y ago

The story of the Boeing engineers flying on the test flights is a perfect example of "skin in the game" (from Antifragile by Nassim Nicholas Taleb). Here's what I wrote about that in a blog post on Antifragility and SW development: At the end of the book, there is a chapter on ethics that Taleb calls “skin in the game”. To have skin in the game, you should share both in the upside and downside. Taleb quotes the 3,800 year old Hammurabi’s code: “If a builder builds a house and the house collapses and causes the death of the owner – the builder shall be put to death”. It is interesting to view this from a software development perspective. I have never worked on software where people’s lives were in danger if the software failed, but I would not be willing to submit to Hammurabi’s code if I did. But I think a little less extreme form of skin in the game is actually very good. Being on call for example. If the software you wrote fails, you may get called in the middle of the night to help fix it. I have been on call at most of the places I have worked in the past, and I think it has a lot of benefits. It gives you an incentive to be very thorough, both in development and testing. It also forces you to make the software debuggable – otherwise you yourself will suffer. Another way of introducing skin in the game is dog-fooding – using the software you are developing in your daily work. I have never worked on software that we have been able to dog-food, but I think that is another great practice. http://henrikwarne.com/2014/06/08/antifragility-and-software-development/ http://henrikwarne.com/2014/06/08/antifragility-and-software...

dbdr 11y ago
The examples you are giving are all about sharing the downside. What are the options for the upside?

crucini 11y ago

I think the right answer here is to develop languages or libraries that enforce the constraints you want. I think we should beware of adding layers of "silver bullet" technologies that promise to fix the last crisis, but increase the friction of development. To illustrate what I mean, PLCs are normally programmed in ladder logic. From that level, I don't think you can crash the machine or corrupt memory. So the risks are limited to the "operating system" if you will, which can be more mature and tested than the "application".