10 ms·
Boeing's software fix for the 737 Max overwhelms the plane's computer
- zxcb1 7y agoCould this be considered unacknowledged technical debt?
- mbar84 7y agoMaybe we're just ignorant of the available chips for this use case. Otherwise just reads like they've been kicking the can of system redesign/refactoring down the road for decades. It appears that short term thinking abounds in every industry.
- zxcb1 7y agoSomething like that, thinking about it as a kind of senescence, arriving at a point where they realize that the system is depleted. It needs to be planned for.
- klingonopera 7y agoI think if Boeing (or Airbus - what do they use? Faster chips or better algorithms or ...?) really wanted that extra processing power, they would've pushed to get the next chip in line to be certified for aerospace applications. The fact that they neglected to do this... I don't know, public perception and Boeing's own representation seemed to paint a picture of "computer" and "chip"-aversion up to the point when they no longer could ignore the issue by designing a plane that needed to have it in order to be certified they way they wanted it to. And by then it was of course already way too late to certify another chip... Then again, on the other hand: Boeing designs a lot of military planes, too, aren't these almost always exclusively fly-by-wire? Shouldn't they have the know-how in these things as well? Or is there a no information-exchange policy in effect between their military and civilian teams?
- fit2rule 7y agoI'm a developer with extensive experience in SIL-4 programming, specifically for life-critical systems (rail transport). I've built software subsystems that are used in 38 countries around the world to keep the trains safe. The issue is the extensive costs of changing from one CPU type to another. Certification for these systems is a multi-year process and can cost millions of dollars even before any kind of success is guaranteed. There are still very old CPU's out there, functioning just fine for decades. Safety programming doesn't just swap out parts like consumer computing does - it takes a lot of work to change CPU's. I don't think Boeing has been kicking the can down the road on the upgrade. I do think they've been trying to cut costs and exploit their customers by offering extra safety features as upgrades, rather than making them standard. Its interesting to note that some of their customers don't require such stringent safety features in the regions they operate - i.e. this is as much of a legislative issue as anything else. It could very well be that Ethiopia doesn't have the same safety requirements encoded in its laws governing flight as France does, so Boeing offers different features not just according to budget but also legislation - although we are sure to see that change rapidly now.
- rob74 7y agoDefinitely - it's important to keep in mind that from an engineering perspective, the 737 has probably already reached EOL, and Boeing has been considering to develop a replacement for years (Boeing Y1 - https://en.wikipedia.org/wiki/Boeing_Yellowstone_Project https://en.wikipedia.org/wiki/Boeing_Yellowstone_Project), but that was delayed until Airbus forced their hand with the A320 neo, so they decided to offer a (as it now looks) "quick & dirty" upgrade of the 737 to keep it competitive. So upgrading the flight computers was probably out of the question with the given development time.
- bcaa7f3a8bbc 7y agoSimple question: Why doesn't Boeing upgrade to i386 for their new planes? I heard it's available for aerospace applications.
- crocal 7y agoI don’t have evidence, but most likely, it is to avoid the related costs and risks of certification of this « new » CPU for its intended usage. Not only the CPU but all the software stack needs to be re-proven safe, including compilers and test tools. Note: this is not necessarily a stupid decision.
- rbanffy 7y agoA simpler solution would be to use the same CPU done with a slightly newer process to allow a higher clock rate. That would ease the recertification process. If you think they are at best 40 MHz 286 parts, the best that was available by then, running software derived from previous releases that ran on slower chips, there is plenty of room for improvement. IIRC, you can get a modern 65816 (the 16-bit descendant of the 6502) in 300+ MHz. And I wouldn't expect it to be on a cutting edge process. That considered, it's entirely possible most of the time is being consumed by reading the sensors. In that case, the sensor communication would need to be upgraded and recertified.
- bcaa7f3a8bbc 7y agoYes, I'm familiar with second-source 6502. Also, I've seen the Rabbit2000 processor, it's basically a Z80 running at 50 MHz. But I wasn't aware that 80286 has any modern second-source clones. Are there?
- rbanffy 7y agoI don't know, but it wouldn't be too hard. Or expensive, in relation to bleeding edge semiconductors. Fun thing: I remember a 40 MHz 286 desktop at one point. Wikipedia tells me they only went to 25 MHz. That's some overclocking... Fun fact #2: with a feature size of 1.5 µm, one could fit an entire 6502 done in 7 nm process on top of a single 80286 transistor.
- aneutron 7y agoThis smells like more lives are about to be lost if Boeing doesn't get their act together
- bradknowles 7y agoNo. A few executives might shave a fraction of a penny off their multi-million dollar annual bonuses. And a lot of lower level workers are going to lose their jobs. But no more lives will be lost if Boeing fails to consolidate their defecation on this matter.
- majewsky 7y agoMods: Can you remove the #more from the submission URL? The fragment makes the browser scroll down which skips the first half of the article.
- dang 7y agoSure.
- dba7dba 7y agoFrom the article ----------- Boeing says that it can again fix the software to avoid the problem the FAA just found. It is doubtful that this will be possible. The software load is already right at the border, if not above the physical capabilities of the current flight control computers. The optimization potential of the software is likely minimal. MCAS was a band aid. Due to the new engine position the 737 MAX version had changed its behavior compared to the older 737 types even though it still used the older types' certification. MCAS was supposed to correct that. The software fix for MCAS is another band aid on top of it. The fix for the software fix that Boeing now promises to solve the problem the FAA pilot found, is the third band aid over the same wound. It is doubtful that it will stop the bleeding. The flight control computers the 737 MAX and NG use were developed in the early to mid 1990s. There are no off-the-shelf solutions for higher performance. Boeing's latest announced time frame for bringing the grounded 737 MAX planes back into the air is "mid December". In view of this new problem one is inclined to ask "which year?" ----------- Ouch... Introducing a new CPU into airplane like 737 will require another whole series of test/certifications. And right about now, Boeing management is thinking we could use some of those senior engineers we laid off because we thought they were not needed as our products are now mature.
- kijin 7y agoDamn, I hope those senior engineers charge Boeing an arm and a leg when they are asked to come back.
- menckenjr 7y agoWhat's the over/under on the executives and senior management who made that decision getting fired for it?
- bradknowles 7y agoWhy should the executives suffer any repercussions? It’s not like the wrote any of the doomed software or anything. They desperately need their multi-million dollar bonuses so that they can afford to continue to live such lavish lifestyles. What’s wrong with you people? You act like a few lost lives have some value or something..... /s
- mongol 7y agoIs the 80286 still in production? Or is it old stock on a shelf somewhere that is used?
- ChuckNorris89 7y agoAFAIK, Airbus has stockpiled huge amounts of legacy CPUs in climate controlled conditions. Boeing must have done the same
- pratap103 7y agoWas really curious as to what Airbus was using, thanks for this! Why not just use the latest CPUs though? I can't imagine the cost would be relatively significant in the greater scheme of things. Are the legacy CPU's really better for this use-case?
- ChuckNorris89 7y agoAero industry moves at a very slow pace. The state of the art F-22 Raptor uses intel 386 cpu and the F-35 lightning upgraded to more "modern" PowerPC CPUs, similar to the ones on legacy Macs. Planes don't need insane multitasking processing power like our smartphones or PCs. They mostly do signal processing and sensor fusion in a tight loop which is quite trivial even for legacy CPUs as it's basic flight math equations which results in highly optimized code. In terms of aero chips, basic is always better as you want a silicon that's tried and tested for decades to have a deep understanding of it's quirks and bugs so you know the code execution is reliable.
- varjag 7y agoThe redesign and compliance costs are enormous. That said, these costs are something Boeing could afford many times over. Manpower could also be a bottleneck. Design talent is not what it used to be in the first world after much of production moved overseas.
- jiberwarrior 7y agoProbably minuscule when stacked next to the costs of grounding all 737MAXs for half a year and the negative publicity of multiple planes catastrophically nosediving out of the sky
- nercury 7y agoDoes this mean that one of the reasons MCAS relied on a single sensor could be the already saturated CPU? Mindblowing.
- laythea 7y agoI thought the extra sensor was an optional upgrade?...
- pritambaral 7y agoNo, that was the AoA indicator in the cockpit. The second AoA sensor was always there, MCAS just didn't consult it.
- tanakachen 7y agoAs has been discussed on HN, this whole 737 MAX flaw is not a software problem. Boeing wants you to believe it is. The flaw is actually a physical design problem of the aircraft. Even if the software were to work perfectly, the plane is still flawed. The only real fix is not to fly the 737 MAX.
- pratap103 7y agoOr admit it's a hardware and aircraft design problem and go through the exponentially more expensive process of fixing it.
- tanakachen 7y agoWhat’s that old quote... “It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It.”
- liberte82 7y agoThat explains so much.
- ChrisRR 7y agoIt's a problem with the entire process. The hardware, software, safety as an option, lack of training, risk assessment, design, alarms that aren't easily identifiable, features that can't be over-ridden. There's issues throughout practically every step and they want people to believe that they're just going to push a software update and everything will be 100% fixed.
- hef19898 7y agoYeah, it seems that Boeing hit a hard design limit with the 737 airframe. Main issue is the low ground clearance which makes it impossible, one could argue, to fit new engines. Which would mean that the 737 reached its end-of-life. That must scare the shit out of Boeing execs, and rightly so when your bread and butter product needs to be replaced by something new.
- 7y ago
- stunt 7y agoThey may find a physical fix for it later. Like adding weight where it can help. Even then, perhaps nobody wants to fly with it plus it can't offer the same performance.
- laythea 7y agoIt amazes me how Pilots are required mainly for the purposes of being able to act as a last resort against a computer which has gone haywire, and the public feel better thinking that the pilot can "take control" and then at the same time aircraft manufacturers are removing the ability for the said Pilots to override the computer. That's progress!