18 ms·
Boeing changing Max software to use two computers
- penglish1 7y agoAnd what happens when the 2 computers disagree? I thought 3 was standard practice for this sort of thing.
- tzs 7y agoYou need 3 for things that if they stop working the plane crashes. This is more of a "if it stops working the pilots have to fly more conservatively until it is fixed" thing. It apparently is generally considered acceptable for such things to not have triple redundancy, as long as a failure will be detected so the pilots can deal with it. Two computers is good enough for that.
- semerda 7y ago“Only one computer was used in the past, because Boeing was able to prove statistically that its system was reliable, the person said.” — here boss no need for redundancy coz we write amazing code that’s statistically never going to fail! What were they thinking?! There is no system 100% efficient. Even a small error rate over the standard ~20yr service life isn’t a guarantee a failure won’t occur. Considering the max are refurbished 737s, wouldn’t more precaution be taken?
- alkonaut 7y agoThe question is whether is fails safely. The design likely considered failure to not include near-inevitable crashes, but rather some manual intervention. In that case N failures per X thousand flight hours is nothing unusual. The change isn’t to get zero failures but to get a system that fails safely. Compare to a Tesla autopilot: it can work 99% of the time, if at 1% of the time it slows down and gives up. It can’t crash into oncoming traffic 1% of the time. The failure mode matters.
- tatersolid 7y ago> Considering the max are refurbished 737s, wouldn’t more precaution be taken? Correction needed: every part in a 737-MAX is new; they are not refurbished older 737s. The MAX are supposed to be brand new “fly-alike” planes that makes replacing older 737 models with something more efficient easier for the airline.
- ilaksh 7y agoThe language in the article seems to conflate hardware and software changes. If they are introducing a new computer, that is a hardware change in addition to software.
- Scoundreller 7y agoI think they're running the software in a second (existing) computer. Which still doesn't make sense from a redundancy perspective: What do you do when 2 systems disagree? Which one is right? I guess you could shut it down, but would be better to have an odd number of systems and take a majority result while displaying an error light.
- jatgoodwin 7y agoFrom what I understand the flight computers can tell the pilots something is wrong and/or the pilot and co-pilot can have their instruments from different sensors like airspeed so they can compare and figure out which one is right.
- Scoundreller 7y agoIn isolation, yes, but hopefully it's not the 4th issue on top of 3 issues the pilots are already trying to figure out.
- jatgoodwin 7y agoThere's already 2 computers, they just don't error check with each other. It's one of the core issues that caused the crashes.
- ekianjo 7y agoRedundant computer is already mentioned in the article.
- teh_infallible 7y agoKeep talking about the software. Make it about the software, so no one knows the airframe is fundamentally flawed..
- artsyca 7y agoA Faustion proposition -- they're going to release software updates from here to the apocalypse yet none of them will succeed in winning back anybody's trust; they should implement a code freeze until hell freezes over
- rrss 7y agoHow do you know the airframe is fundamentally flawed? ("the engines are too big" is not an answer).
- dboreham 7y agoThe wings are too low?
- notjustanymike 7y agoKeeps crashing into the ocean
- toomuchtodo 7y agoIf the airframe isn’t flawed, fly it without MCAS. With Boeing management onboard. Surely you don’t need to dynamically adjust flight dynamics without the pilot’s input or knowledge if the airframe is sound. It’s realistic to build aircraft where software is required to fly said aircraft depending on the flight dynamics, and the pilot(s) are fully aware of what’s running. But that changes the type rating and blows Boeing’s regulatory sidestep out of the sky.
- TylerE 7y agoGod it's frustrating seeing people here talk about aviation. Flight computers are a fundamental part of normal aeronautical design. Asking Boeing to fly a Max without MCAS would be like asking you to write a bug free C++ program... with a compiler with no error messages or validation.
- qaq 7y agoThis sounds pretty scary a major rewrite on tight schedule under huge pressure.
- doggydogs94 7y agoHopefully, 99% of the new code was already written, not something they just whipped up and are slamming into production at the 11th hour.
- Chris_Chambers 7y agoI think we can rest assured that it's definitely something they just whipped up and are slamming into production at the 11th hour.
- nemild 7y agoFrom the article: >“Only one computer was used in the past, because Boeing was able to prove statistically that its system was reliable, the person said.” Years back, my father co-wrote a paper on the Space Shuttle Challenger showing issues with the statistical thinking at NASA: http://www.math.montana.edu/shancock/courses/stat401/Dalal_etal_1989-Challenger.pdf http://www.math.montana.edu/shancock/courses/stat401/Dalal_e... I wonder what statistical techniques Boeing used, and how defensible those techniques were.
- voldacar 7y ago"Prove statistically" is such an odd phrase. I wonder if they used fuzzing or something like that? Because even so that is quite far from a formal software proof
- jwilliams 7y agoIt's relatively common in embedded programming (in my experience). A lot of real-time programming is around scheduling. You certainly can use formal proofs there too, but statistical methods would be common at significant scale.
- na85 7y agoIt's quite common in aerospace certification to cite systems having proven records of an acceptably-low number of failures per flight hour or similar metric. I suspect they're using that sense of the word "proven", as opposed to a formal software proof.
- pdpi 7y agoDon’t know what’s going on with their certification process here but probabilistic reasoning is very common, and very reasonable, eg in cryptography.
- danjayh 7y agoFor those who are outside of the industry, the article is probably not completely accurate. I don't know specifically about the 737 MAX, but for many of their other newer airframes (787, upcoming 777x), Boeing relies on a concept called 'high integrity at the source'. Essentially, two complete copies of the flight computer hardware are put on a single card and they cross-compare their results. If you're looking for a bit of dense reading material on the subject, you might find a related patent application interesting: https://patents.google.com/patent/US9170907 https://patents.google.com/patent/US9170907
- deleted 7y ago[deleted]
- tracer4201 7y agoI’ve lost all faith in Boeing. Boeing’s greed and cutting corners to avoid pilot recertification is textbook example of ethical failures. I have second thoughts about many of the companies I’ve invested in over the years, but Boeing right now takes the cake. I’ve dumped my shares. Hopefully there are some criminal investigations at some point if our leaders in Washington could stop dancing to the tune of whichever company throws them a bone like the dogs they really are.
- skunkpocalypse 7y ago> I’ve lost all faith in Boeing. Guess what? They don't care because it doesn't matter. You really can't avoid their airplanes without avoiding air travel.
- tempguy9999 7y agoIs boeing really the only producer of aeroplanes? My guess is there are others. My guess is that orders will be cancelled and there'll be a shift towards airbus (oh!there's one!) I'm tempted to apply my first ever downvote to this apparently witless comment, but perhaps you meant something more subtle?
- _ph_ 7y agoOrders can't just be shifted to Airbus, as their production lines are fully loaded for years to come. Also, there is little desire to make Airbus a monopolist. The problem basically started when Boing was merged with McDonell Douglas, as this created the duopoly which now has the market trapped. And of course it creates a huge amount of problems for all the airlines which ordered and planned for new planes which have the same certificatation as their existing 737.
- JohnJamesRambo 7y agoIt feels like we see this over and over when you trace corporate problems back- loss of competition and development of monopolistic conditions. There’s a reason we developed antitrust laws and to watch them be eroded in the modern era is quite worrisome.
- wayanon 7y agoWill anyone appear in court for this episode in a year or two?
- AnssiH 7y agoA Seattle Times article from a few days ago seems to contain more detail: https://www.seattletimes.com/business/boeing-aerospace/newly-stringent-faa-tests-spur-a-fundamental-software-redesign-of-737-max-flight-controls/ https://www.seattletimes.com/business/boeing-aerospace/newly... It includes e.g. a description of FAA bit-flip testing to induce incorrect system behavior that Boeing intends to solve by using two computers.
- danjayh 7y ago"The fault occurs when bits inside the microprocessor are randomly flipped from 0 to 1 or vice versa. This is a known phenomenon that can happen due to cosmic rays striking the circuitry." This is called Single Event Upset. For all those of you that aren't in the industry, essentially the problem is that any bit inside the flight computer (RAM, cache, NVM, registers - ANY bit) can change state randomly at ANY time. It's rare, but when you get into millions of flight hours, it WILL happen. The software and hardware have to be designed to mitigate problems caused by this behavior.
- GistNoesis 7y agoIf this is due to cosmic rays, doesn't flying over the poles make it more likely that such event happens ? How are we sure there aren't local bursts of cosmic rays that would make suddenly a few of those Single Event Upset, for example when there are some high likelihood of seeing northern lights ? How do you test your hardware and software to show that you are indeed cosmic-ray proof ?
- fit2rule 7y agoYou test the living crap out of it, and not just in the lab on the workbench but also in operation while online - while the thing is running in operation, it is also consistently testing itself to ensure that the hardware is performing as expected. Online software tests check for cosmic ray bit flips about 1000 times a second, in addition to whatever hardware mechanisms are in place to detect this (ECC, etc.) This is a standard module in most SIL-4 applications, where 2 of 3 consensus model is being used. What I don't understand is why Boeing aren't using 2-of-3 computer architecture in this application - or maybe they are, and the '3 voting units' are considered to be 'one computer' and they've just added another one to be sure. In rail transportation systems, this is taken even further by using 2-of-3 configurations where each computer is a different architecture completely ..
- Randor 7y agoMany years ago I wrote navigation software for ocean going vessels/ships. We used double and triple redundancy on many of our sensor types. We generally used three control computers that would 'vote' before deciding to make vessel navigation changes. We also always included a "Dead man's switch" that allowed the bridge crew to take control at any time. I can't even imagine designing aerial vehicle autopilot without redundency. The stakes are too high... I would be interested in having a look at the statistical model they used to prove 'the system was reliable with zero-redundancy'. While designing these systems for ships the only way we were able to get an error probability near zero was when we used triple redundancy.
- adreamingsoul 7y agoI recently watched a documentary about a diving ship that had a tripple redundancy positioning system that failed during a dive. It was fixed with a hard reboot.
- inamberclad 7y agoIt wasn't an autopilot that failed. This was just supposed to be an accessory system.
- rkagerer 7y agoHow did you deal with redundancy "choke points"? eg. What if the component that tallies computer votes and actuates things based on the results fails (especially in a way that's hard to detect)? Were you able in some cases to maintain isolation all the way through from sensors to actuators and design such that a single failed one (in a worst case failure mode) could be overcome by the rest?
- fit2rule 7y ago(Disclaimer: SIL-4 programmer for safety critical rail transportation applications) The way this is done is that 2 of 3 computers need to 'agree' on the final decision in order for it to be considered the correct one - there isn't a single point of assessment, but rather a consensus that must be formed from the results of all 3 computers. Ideal case, all 3 produce the same results. This has worked successfully for decades. What's changing now, is that those 3 computers are now no longer the same architecture - you'll have a PPC and an x86 and an ARM-based CPU all attempting to agree to the same data, in order to prevent systemic failure throughout.
- ggm 7y agoHas Boeing now lost enough money, that had they not tried 737 certification they might be financially ahead?
- Aloha 7y agoWithout certification as a 737, the airplane wouldn't have much of a reason to exist.
- inamberclad 7y agoPeople here seem to be missing a few things. 1: The 737 family, including the 737 MAX, still use manual/hydraulic controls, not fly-by-wire. A computer failure isn't supposed to be as catastrophic as it would in a fully fly-by-wire system. 737 pilots are already trained on other automatic system failures, such as a pitch trim motor runaway. 2: The original design of MCAS had a limited scope - stop pitch instabilities from causing an uncontrolled pitch-up into a stall in high AoA and high power regimes, such as just after takeoff. It read from both a G-sensor and the AoA vane and both sensors had to be reading excessive values for the system to trigger. This was probably the justification for not requiring higher levels of reliability. 3: The plane was found to have handling issues in slow flight regimes. In order to improve the handling, Boeing engineers modified MCAS to be active in more of the flight regime, removing one of the sensor readings. Now it would trigger if it detected the plane was near a stall, even in level flight without a high G loading. It is unclear whether MCAS was recertified within Boeing for continuous use. Wikipedia states "The FAA did not conduct a safety analysis on the changes. It had already approved the previous version of MCAS, and the agency's rules did not require it to take a second look because the changes did not affect how the plane operated in extreme situations." 4: Pilots could not disable MCAS without disabling electric trim control. In at least the LionAir case, the pilots did disable the electric trim, but were unable to re-trim the plane manually against the aerodynamic forces of their dive. They re-enabled the electric trim to re-trim the aircraft, and MCAS re-triggered and put them back in the same situation. Remember, these aircraft use manual controls. The pilots need to put a serious amount of force into the control column when the plane is out of trim and at a high airspeed. This greatly diminishes their ability to do anything else. 5: Both accident aircraft were missing an optional safety feature - the AoA disagree warning. Both aircraft experienced a failure of one of their AoA vanes. I don't recall whether these failures happened in flight or before the flight, but such a warning would have likely stopped the pilots from taking off. 6: The single computer issue was not uncovered until simulator testing after the crashes. It is not directly related to the MCAS problems. Wikipedia is your friend, as usual. https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au... Here's even more detailed information: http://www.b737.org.uk/mcas.htm http://www.b737.org.uk/mcas.htm
- 7y ago
- josemanuel 7y agoI don't understand... so the crashes were due to faulty single sensor. Adding computer redundancy won't fix single sensor screw up. Will they retrofit all planes with dual sensors?! I really don't want to fly on this airplane. This whole thing just sounds like PR trying to sell the idea that this plane will be fit to fly when it will clearly come back into active with a heap of flaws...
- Filligree 7y agoAIUI, all the planes already have dual sensors. So they wrote the software to use one at a time, alternating at each take-off...
- dTal 7y agoOf all the broken design decisions in the MAX, this was the one that elicited an outraged "what?!?" from me. In what possible way could this be a good idea?
- keymone 7y agoA tiny additional detail - sensor alternates not because of some arbitrary adhoc rule but because aircraft always uses sensor on main pilot’s side and main pilot’s seat alternates each flight. Still boggles my mind how utterly incompetent and borderline malicious everybody involved in designing and implementing this system are. I wish we’d see some people jailed and Boeing hit with billions of dollars fines but chances are slim.
- dTal 7y agoI am aware of that detail. But why? Why does the aircraft always use the sensor on the main pilot's side? That itself seems like an arbitrary, ad-hoc rule. What does global AOA state data have to do with where the pilot's ass is currently located? Why even have two sensors at all, if you only use one at a time? Given you have two sensors, why not fuse them? Any fusion logic is more intelligent than what they did. I'm sure each decision seemed logical to the person making it. But from a bird's eye view, it gives the impression of poor coordination - Team A [hardware] thinks "clearly we need multiple AOA sensors for redundancy". Team B [software] thinks "AOA isn't all that important, we barely use it and failure is non-critical. Sensor fusion is hard, let's go shopping!".
- alkonaut 7y ago>“Only one computer was used in the past, because Boeing was able to prove statistically that its system was reliable, the person said.” I read that as ”statically” (thought it meant formally) and was really impressed by how advanced that sounded! Then I read it again and saw statistically. Less impressive.
- Tomte 7y agoThe question of statistical reliability is also an American-European (or at least -German) divide: if you look at machinery safety, the old German standard 13849 is all about architecture. Two channels, two channels with diagnostics, etc., yes, also a bit of MTTF numbers, but architectural concerns are central. The American way is using statistics to argue that something is not dangerous in a meaningful way (for example several magnitudes less likely than being hit by lightning). There is even a white paper by ABB arguing that safety controllers should be allowed to be purely one-channel architectures, because reliability is so good. Most people in the field would take that as humorous. And so the "modern" standard 61508 is a bit of a hybrid. There are still architectural elements (hardware fault tolerance numbers etc), but front and center is also statistics, namely the SIL levels.
- redis_mlc 7y ago>“Only one computer was used in the past, because Boeing was able to prove statistically that its system was reliable, the person said.” Yeah, I have a real problem with that. A priori of a computer failure, you may have statistical reliability. But once it actually fails, the failure rate is 100% and a backup computer is needed. Also, external AoA sensors fail all the time (like when a bird hits them), so the failure probability is very high. So we're not just talking about solid-state computer reliability here. What also concerns me is that airliners fly at transsonic speeds. How would the AoA calculations be affected when the airplane is in a slip for some reason, and the AoA sensors are reading different values? What happens when MCAS goes berserk at high speeds (ie. near Mach 1.) If MCAS wasn't even tested on takeoff, it sure as hell wasn't tested at Mach 0.8. If the stakes are not clear ... besides the loss of life, an airliner crash often results in the bankruptcy of the entire carrier. Source: commercially-rated airplane pilot who's studied high-speed aerodynamics.
- crocal 7y agoStatistics are used all the time in safety critical designs. But, this is key, they are then balanced with the outcome expected due to each statistical failure. The more serious the outcome, the less the tolerated statistical failure. In the case of MCAS, from what we have read so far, it seems this analysis was biased toward a non-catastrophic outcome (e.g. manageable through pilot action). This may explain why the statistical failure of a single computer was deemed sufficient. Now, assessors have upped the stakes, and the stats do not work anymore. This indeed reveals a very serious problem in the RAMS process at Boeing, and exceed the scope of a « simple » software fix.
- varjag 7y agoAs an embedded systems developer, I have developed mistrust to statistical reliability figures. These days you quite often come across components and subassemblies with MTBF stating (sometimes tens of) millions of hours. When you try to clarify with the vendor how they arrive at these figures there is nothing but general handwaving about using industry tools and statistical models. But sorry, just can't seriously take it on faith that your product is going to last 11 centuries at 20C.
- meerita 7y agoI'm surprised they 3rd party some parts of their software. Software is it the most important part of a plane, how crazy you should be to do that.
- pankajdoharey 7y agoWoah thats a standard even in tesla cars. Everyday it becomes more apparent that Boeing is a disaster engineering project.
- bfrog 7y agoThis seems like a pretty significant change, something that won't take a few months but possibly years of work to do?
- ypcx 7y ago> Boeing changing Max software to use two computers Does that mean we will find the MCAS in the 737 MAX 8 MMEL (Master Minimum Equipment List)? (Used to be here but right now it's not loading: http://fsims.faa.gov/wdocs/mmel/b-737-8_rev%200.pdf http://fsims.faa.gov/wdocs/mmel/b-737-8_rev%200.pdf) Isn't this the single biggest failure point of the whole affair? Adding a critical component which can bring the plane down (as it has happened twice) but not making it redundant? One can perhaps argue (pardon the roughness of my analogy) that if pilots aren't trained to not open the door during the flight, and then do open the door during the flight, then it's a training issue. But this analogy doesn't apply, because while the door is not motorized and cannot open by itself, the MCAS is a computer system which can fail (not just because of a faulty AoA sensor, but e.g. because of a chip failure or a software bug) and then actuate the trim into a dive where the pilots aren't able to fix the issue manually anymore due to high wind pressure.
- redis_mlc 7y agoHow the MMEL (FAA) and MEL (operator model-specific version) works is that anything NOT required for safe flight is listed in the MEL. Otherwise everything would have to be 100% working. So I would not expect MCAS to be listed in the MEL since it's required for safe flight. We'll see.
- ypcx 7y agoOh, I've gotten the MMEL definition wrong then (opposite). Thanks for clarifying!
- nurblieh 7y agoAnyone who has watched "Minority Report" or "Neon Genesis Evangelion" knows you need 3 computers.