8 ms·
> Because the fault was triggered by specific streams of erroneous flight data, a new software patch can be devised that monitors the computer for that highly u
by pcl 7y ago
> Because the fault was triggered by specific streams of erroneous flight data, a new software patch can be devised that monitors the computer for that highly unusual condition and prevents movement of the stabilizer when it occurs, one of the people said.
Oh man. That sounds like a mess.
- zerogvt 7y agoWhen your design is a mess you're probably better off to scrap it altogether and start from a clean piece of paper. "Fixes" to a bad design are adding to the mess. And in this case the mess has already cost hundreds of lives. You cannot "patch" this...
- DanHulton 7y agoI see this sentiment a lot, but its kind of surprising. Traditional wisdom states the opposite - never do a full rewrite, you lose the years of buit-up fixes and edge cases and have to learn from scratch in too many places. Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety. Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.
- acqq 7y agoThe problem here is that what Boeing is selling ("737 that spends significantly less fuel and doesn't need recertification") simply can't be made safe. Whatever results from the marketing driven goal (selling something that doesn't and can't exist in reality) will remain unsafe as long as all the "already sold features" are preserved. So it's not if they should "start from clean" but just accept the need for the new certification and then do the changes that are really needed. It will be less profitable for Boeing but it righly should be this time! They are killing people otherwise.
- selectodude 7y agoThe funny thing is this plane already exists. It’s called the 757 and Boeing stopped making it 15 years ago because the airlines wanted stretched 737s far more than they wanted the clean sheet redesign.
- teawrecks 7y agoBandaid fix if I've ever heard of one.
- User23 7y agoMy jaw literally dropped. It actually took me a moment to realize I was sitting gape-mouthed reading that. We're talking airplanes here, not some LAMP stack. If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.
- Aloha 7y agoWe effectively have a front row seat to the sausage making of software development, its not pleasant.
- ulfw 7y agoThat describes the whole 50+ years old 737 to it’s core.
- burfog 7y agoIt's proper. This is exactly how you do things with life-critical embedded software. You detect when things are crazy, which of course shouldn't happen but nevertheless does, and you react in a way that is hopefully not disastrous. The alternative is to ignore the problem. Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad. When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement? Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.
- Haga 7y agoOnly embedded guy here. You can't catch them all. Height radiation alone sees to that.
- aurelwu 7y agoThat sounds like adding a heuristic which works most of the time which is something I can see myself doing to cover some pathfinding issue in a game where it's sufficient if it works 99,9% of the time or when working on a simulation where I can look on the result data afterwards and can see if something looks highly unusual despite the heuristic and improve it iteratively - a luxury you don't get with air planes, at least not without casualties. Additionally given the very limited computing power of the hardware in the 737 adding even more software to it might create even more problems.
- thelittleone 7y agoWasn’t aware the compute power was low on the 737. Interested if you have some more info? Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?
- Maakuth 7y agoHere's an article about it. According to this piece, 737 avionics run on a pair of 80286s. https://www.moonofalabama.org/2019/06/boeings-software-fix-for-the-737-max-problem-overwhelms-the-planes-computer.html https://www.moonofalabama.org/2019/06/boeings-software-fix-f...
- seymore_12 7y agoThe should at least get 1202 error..
- tyingq 7y agoIf true it uses a 286, for some relative comparison, the fastest 286 is 1.5 MIPS. A run of the mill current Arduino is probably 20 MIPS, and a Raspberry PI3 is 2451 MIPS.
- robin_reala 7y agoCan’t find the exact MIPS estimate for the Harris CS80C286 25Mhz 286, but given that the 1.5MIPS was quoted for the 10Mhz Intel one I’d guess it’s substantially faster. Although still slower than an Arduino :)
- xvf22 7y agoIt's one of those fixes where they add up the value of human life if something goes wrong and say 'OK we can handle that, ship it"
- alexis_fr 7y agoI wonder if the human life calculation worked well this time. As far as I see, Boeing lost more than the sum of the human lives; they also lost reputation for everything new they’ve designed in the last 7 years being corrupted, and they also engulfed the reputation of FAA with them, whose agents would fit the definition of “corrupted” by any people’s definition (I know, they are not, they just used agents of Boeing to inspect Boeing because they were understaffed), and the FAA showed the last step of failure by not admitting that the plane had to be stopped until a few days after the European agencies. In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.
- lizmat 7y agohttps://en.wikipedia.org/wiki/De_Havilland_Comet https://en.wikipedia.org/wiki/De_Havilland_Comet
- iguy 7y agoExcept that, unlike de Havailland, Boeing is too big to fail. There's just no way the US will allow it to keel over, and leave Airbus the sole supplier. And as with the banks, you can bet that this implicit guarantee gets figured into what decisions people take.
- jacobush 7y agoThere is so much of U.K. which is incredible and depressing at the same time. Completely pummelled by war, (receiving no aid like West Germany did) they managed to, in 1949, release a Space Age passenger commercial airliner, defining the genre. It looks modern even today. Yet, fumbled the ball. They also had a space program, closed it down after their first satellite, which was a success. Concorde! (With France.) It goes on an on, I'm sure you have favourite examples. It's heart wrenching.
- tus88 7y agoWow a textbook example of a bandaid solution. Reminds me of something a novice programmer would do. When faced with a complex problem....nothing beats an even more complicated solution!
- goodcanadian 7y agoSounds like a running joke at my work. Any hardware problem or design failure: you can fix it in software!
- zwaps 7y agoDo you work at Boeing? ;}
- rasz 7y agoits industry standard https://en.wikipedia.org/wiki/Application_firewall https://en.wikipedia.org/wiki/Application_firewall https://en.wikipedia.org/wiki/Web_application_firewall https://en.wikipedia.org/wiki/Web_application_firewall
- minedwiz 7y agoTo be fair, any hardware running a WAF in Prod these days is going to have more compute power than a 737.
- heisenbit 7y agoYep. At this point in time they should have plenty of test cases and regression tests which should catch such situations. While Boeing may feel they are doing a lot of testing in the simulator - there are way fewer simulators staffed with test pilots than there are 737 MAX out there. When such errors surface at this time in manual testing it calls into question the development process. And a lot of the trust - aka. certification - is based on that process. As I have mentioned here before: The angle of attack sensor was never intended for the purpose it is now used for - military aircraft may do such things but not as an afterthought and not connected with a control surface which has a lot of delay (trim wheel winding. Delay is a major factor in destabilizing control loops). Equivalent sensors are the airspeed pitot tubes but they are a much more robust concept. Even they have brought down airplanes. We lack sufficient real world (heat, humidity, snow) information about the quality of the sensor data even before it may get distorted by transmission and processing on its way to the control law. Cleaning and validating sensor data is tricky under the best of circumstances. It does not sound like they have a handle on it. Hi Boeing, what real world data from mounted angle of attack sensors have you analyzed? Why do you believe this data sufficiently represents climate zones and flight conditions (take-off, landing, turbulence, malfunction modes)? How do you clean and validate the sensor data? Please explain, because otherwise I won't step into one of these machines.
- michaelcampbell 7y agoCan you forward the data Airbus, your car or public transport vehicle manufacturer, Otis or equivalent elevator company obviously sent you for their respective operating environments?
- laythea 7y agoIt is. But don't worry, when it is found out that it is a mess, we have a plan. We can add a software patch to correct the bad software output. Not fix the bad software. But add some more to correct the output. Genius!