3 ms·

The saddest thing, as many HN users should know all to well, is that there can be no excuse for automated systems like airliners to experience catastrophic fail

The saddest thing, as many HN users should know all to well, is that there can be no excuse for automated systems like airliners to experience catastrophic failure and loss of life, if only due to the availability and application of modern SDLC principles and CI/CD etc. Smoke testing could have been performed such that all possible combinations of transducer input could be considered and evaluated thoroughly for closed-loop effect at runtime. These types of integration tests should have been performed repeatedly, seemingly endlessly in the quest for bugs and analysis of the full spectrum of runtime results and effects. In my experience in the software industry, I've always done this for applications that have infinitely more trivial effect and results than an airliner at altitude containing hundreds of souls. One potential counterpart to the seemingly infinite greed we see exponentially increasing could be the old adage that karma is a bitch.

philpem 8y ago
Speaking as someone who's done this (though not on a something as big as an airliner!) Yes, you can test control loops -- you can even turn it into a unit test. At least in theory. The problem is that to do the test you need either a working, physical system or a good model. So if you're making a shutdown valve for a chemical plant, you need a physical build of that control valve. Even on that scale, you're talking about something that could potentially fill an engineering lab, be quite noisy and have a considerable amount of stored pneumatic or hydraulic energy. It's possible, but not exactly practical. The alternative is to model the system, but now the question changes: how can you be certain that your model is accurate and models all the variables? Say your valve is slower when it's cold and you don't model that -- now you have a false positive result ("it works" -- but nobody realised that "temperature" was a dependent variable). So you take the middle ground - you can have the test jig for a week, so you record the inputs and outputs for a week under varying software conditions. But those recordings are only valid for that specific timing -- if you change the software and change the timing (maybe you move the trim motor slower), you get a model change and a false positive or negative. It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary.
- torqueTorrent 8y ago
  I agree wholeheartedly and routinely run concurrent intensive smoke tests on real-world HW as well as smoke tests on finely-modeled virtualized environments. Even with the best modeling and virtualization, a true and thorough, 100% 1:1 approximation with the real world at runtime can likely never be attained for a myriad of reasons. However, when lives are on the line, this gap must be closed in some manner so as to provide a greater degree of confidence. Even the most thirsty organizations with lesser consequences for their failures are usually conservative enough and risk-averse enough to know better than to release without thorough (and relatively inexpensive) testing. My old boss used to tell stories about back in the mainframe days whereby he would send customers fancy, branded and shrink-wrapped finished-product but containing blank tapes for the latest release in order to buy a couple of weeks of extra dev time if he thought the software wasn't ready to escape.
- SamuelAdams 8y ago
  > It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary. Agreed. According to other sources [1], management rushed the development work so they could come out ahead of one of their competitors. "But several FAA technical experts said in interviews that as certification proceeded, managers prodded them to speed the process. Development of the MAX was lagging nine months behind the rival Airbus A320neo. Time was of the essence for Boeing." [1]: https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/ https://www.seattletimes.com/business/boeing-aerospace/faile...

philpem 8y ago

Speaking as someone who's done this (though not on a something as big as an airliner!) Yes, you can test control loops -- you can even turn it into a unit test. At least in theory. The problem is that to do the test you need either a working, physical system or a good model. So if you're making a shutdown valve for a chemical plant, you need a physical build of that control valve. Even on that scale, you're talking about something that could potentially fill an engineering lab, be quite noisy and have a considerable amount of stored pneumatic or hydraulic energy. It's possible, but not exactly practical. The alternative is to model the system, but now the question changes: how can you be certain that your model is accurate and models all the variables? Say your valve is slower when it's cold and you don't model that -- now you have a false positive result ("it works" -- but nobody realised that "temperature" was a dependent variable). So you take the middle ground - you can have the test jig for a week, so you record the inputs and outputs for a week under varying software conditions. But those recordings are only valid for that specific timing -- if you change the software and change the timing (maybe you move the trim motor slower), you get a model change and a false positive or negative. It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary.

torqueTorrent 8y ago
I agree wholeheartedly and routinely run concurrent intensive smoke tests on real-world HW as well as smoke tests on finely-modeled virtualized environments. Even with the best modeling and virtualization, a true and thorough, 100% 1:1 approximation with the real world at runtime can likely never be attained for a myriad of reasons. However, when lives are on the line, this gap must be closed in some manner so as to provide a greater degree of confidence. Even the most thirsty organizations with lesser consequences for their failures are usually conservative enough and risk-averse enough to know better than to release without thorough (and relatively inexpensive) testing. My old boss used to tell stories about back in the mainframe days whereby he would send customers fancy, branded and shrink-wrapped finished-product but containing blank tapes for the latest release in order to buy a couple of weeks of extra dev time if he thought the software wasn't ready to escape.
SamuelAdams 8y ago
> It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary. Agreed. According to other sources [1], management rushed the development work so they could come out ahead of one of their competitors. "But several FAA technical experts said in interviews that as certification proceeded, managers prodded them to speed the process. Development of the MAX was lagging nine months behind the rival Airbus A320neo. Time was of the essence for Boeing." [1]: https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/ https://www.seattletimes.com/business/boeing-aerospace/faile...