16 ms·

737 Max: 1960s Design, 1990s Computing Power and Paper Manuals

118 points by bushido 8y ago

pinewurst 8y ago
The "1990s Computing Power" comment is ignorantly harsh. The MAX (to the best of my knowledge) uses a similar AMD29050-based architecture as the 777 (where it was pioneered) and the 737-800/900. It's amply fast and very very reliable, hardware and software (setting MCAS aside which is new to MAX). Any sane person would prefer the solid well-proven choice over the bleeding edge. People used to make similar comments about the AGC and the Space Shuttle GPCs e.g. that such-and-such-a-PC-was-faster. Yeah, so what?
- londons_explore 8y ago
  I'm not sure I agree here. There exist possibilities for far more safety with more compute. For example, when a mechanical failure occurs (for example an engine explodes and partially falls off) the flight characteristics of the plane change dramatically. Current systems fall back to human control, and hope that human will be able to figure out how to control a now very "unique" machine. Future systems could dynamically create new flight models based on collected data to be able to fly the machine in entirely new ways. For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? [1]. That sort of 'fly it how it has never been done before' is out of reach of human control, and might save lives. [1]:. https://www.ethz.ch/en/news-and-events/eth-news/news/2013/12/new-algorithm-makes-quadrocopters-safer.html https://www.ethz.ch/en/news-and-events/eth-news/news/2013/12...
  - exelius 8y ago
    This tech is just now making it into military aircraft in the F-35 (it’s designed to compensate for failures or combat damage while keeping the plane’s handling relatively consistent), so I would guess it’ll be a while before we see a civilian version.
    - AnimalMuppet 8y ago
      "Figure out a way to fly the thing in the new, crippled configuration" is hard. Figure out a way that's better than what the human pilot came up with may be even harder.
      - londons_explore 8y ago
        It's hard, but it's a very measurable problem. You randomly deform the airframe in a physics simulator, then get a real pilot to try to fly the simulated damaged airframe to the nearest airport. You then get an AI to try doing the same. As soon as the AI manages to get to the airport more often than the human, the tech will save lives, on average. My guess is a PhD student could come up with an AI passing the above test inside a year. Yet aviation standards are stringent enough AI will be decades away from production use, if ever.
        TheOtherHobbes 8y ago
        The political problem is that people still trust humans a lot more than they trust machines. So passengers would be nervous about being flown by an AI, even if the AI was safer. There's some weird psychology behind this: human pilots/drivers/etc are seen by passengers as a personal proxy, with agency over any situation. If you take away the proxy you take away the illusion of agency. Humans really do not like being put into situations where they believe they have no agency at all. As AI gets smarter, this will become more and more of a problem, until there some kind of cultural shift because AI is obviously safer it's not even a question any more. This may or may not happen.
        notahacker 8y ago
        Your link to ZTE trumpeting the success of an algorithm helping an actual physical model of a quadcopter with a disabled rotor operating in lab conditions to crashland a bit more softly might be interpreted as a hint that the general case of safely landing a stricken aircraft is the sort of problem that probably isn't solved by a single PhD in a year... Keeping those with enough hubris to say "yeah, shouldn't take long if we've got some sort of sort of simulator to devise scenarios for a neural network to overfit to" about a wide range of failure modes that an entire field has spent decades studying well away from safety critical systems might be one of the more underappreciated aspects of tight regulation.
  - jacquesm 8y ago
    > For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? The linked article does not support your point, it shows a quadcopter losing one rotor which then goes into a kind of 'controlled fall', two (or even three) rotors is never mentioned and even the one rotor missing scenario involves the whole craft spinning (see video on the page).
  - theclaw 8y ago
    There also exists possibilities for far more danger. As you increase compute, you can run more code, and build more complicated interdependent components that are more difficult to test, especially for the sort of unexpected scenarios that "dynamic" behaviour would warrant. You would basically have a whole new plane - the opposite of what the 737 MAX 8 was intended to be.
- wvenable 8y ago
  > The "1990s Computing Power" comment is ignorantly harsh. I don't think so. You build a plane that's expected to run for decades -- if you put a 2010 computer in there now, it'll be 20 years old in 2030. That 1990's computer was put there in the 90's, does that really make it more reliable?
  - pinewurst 8y ago
    That so-called 1990s computer is probably a year old, embedded in a Honeywell module that’ll be available new for the next 15 years.
    - wvenable 8y ago
      Of course. It's not literally a 1990's computer. But the idea that it's just somehow superior because the design is 3 decades old doesn't make any sense. 30 years ago it would have been new technology; so where was that same argument then?
      - deleted 8y ago
        [deleted]
      - bronson 8y ago
        It's had 3 decades of experience and bugfixes.
        wvenable 8y ago
        If new developing is happening, as it was in this case, that doesn't really matter. This problem was amplified by maintaining old technology, not mitigated by it.
        pinewurst 8y ago
        I don’t agree. The failure(s) weren’t in the old stuff, they were in the poor and ultimately unsafe cover-up (for lack of a better word) of the new. The CG change from the new engines specifically, being masked by new software, MCAS. The only old part culpable is the legacy 737 limit of 2 AoA sensors (with only 1 used for MCAS input (a new thing)).
- BuildTheRobots 8y ago
  The "1990's Computer Power" comment is factually correct. According to wikipedia, the AMD29000 processor was first released in 1988 and "In late 1995 AMD dropped development of the 29k..." I'm not commenting on whether that's appropriate to be running your aircraft or not, but it does very much seem to be a 1990's CPU running approx 50mhz.
chmaynard 8y ago
The entire management chain that dictated these design compromises should be surgically removed from Boeing and prosecuted for negligence. Of course, that will never happen.
- PhantomGremlin 8y ago
  This doesn't get said enough. Within days of the Lion Air crash, MCAS was strongly suspected to be the problem. And yet Boeing management allowed 157 more people to die in the Ethiopian Airlines crash. And then the Boeing CEO still tried to tell everyone that it was a safe airplane. The entire top management of Boeing, and the entire board of directors, should be perp-walked out the door. Of course, that will never happen.
kbos87 8y ago
The thing that baffles me - why the hell has there never been any innovation in the form of making it easier to diagnose a problem and move through the necessary steps in a checklist beyond fumbling through a paper manual? In a situation where seconds count, it seems illogical that this has never been improved upon. Didn’t at least one of these planes crash because the pilots are believed to have only gotten X number of steps through a list of possible reasons?
- InTheArena 8y ago
  The entire aviation business and safety structure is build on extraordinary rigidity to processes and approaches. In addition, automation is viewed as a threat to multiple worker groups. Sometimes this is justified -certainly the MCAS incident is a example. Changes like the MCAS should never have been allowed to change a fundamental part of the airplane interface without considerably more processes being met. That said, sometimes opposition to enhancements is trade unionists being trade unionists.
- Qqqwxs 8y ago
  I remember hearing about the US Airways flight that had an emergency landing in the Hudson river (the film 'Sully' is based on this). The plane sunk faster than it should have because the checklist item 'close the vents' was too far down to be completed before the water landing. In response, the checklist was edited and the item was moved further to the top. Apologies for not finding an exact link, but here is the wikipedia article for the flight: https://www.wikiwand.com/en/US_Airways_Flight_1549 https://www.wikiwand.com/en/US_Airways_Flight_1549
- bdonlan 8y ago
  The newer airbuses actually do have electronic checklists including - for at least some steps - automatic detection that the step has been completed. You can see an example of the electronic checklist in this simulator video: https://www.youtube.com/watch?v=jzjyr0UZKRw https://www.youtube.com/watch?v=jzjyr0UZKRw Even on aircraft without fully integrated checklists, many operators have switched to using iPads instead of paper manuals as well.
whoisjuan 8y ago
Kind of unfair. Do they really think that an aircraft should be powered by the same software or hardware that powers a modern computer or smartphone? An aircraft needs incredible reliability. You don’t get that with modern software/hardware architectures.
- da_chicken 8y ago
  Yeah, the B-52, adopted in 1952, is planned to be is service through the 2050s. An airframe design expected to last for over 100 years. Keep in mind that flight itself is barely 100 years old. And paper manuals? Are they expecting to use iPads for in flight documents or something? Shall we compare failure states of paper vs tablets?
  - TylerE 8y ago
    There are still DC-3s from the early '30s in service.
    - Aloha 8y ago
      The old saying is "the only replacement for a DC-3 is another DC-3" Though I think most of them that are flying are C-47's from the war era.
  - tedivm 8y ago
    That article says that other plans will display the error and the recommended checklist automatically, which certainly seems faster than flipping through a bunch of books trying to find a specific page based off of some lights.
    - ReptileMan 8y ago
      A lecturer of mine once said - knowing what kind of people we teach here for programmers, I am scared of going to the doctor. The less software is in something(and less cloudy/iot) the better. And the more I program the more I prefer stupid and mechanical things when reliability is on the line. A paper manual is always there for you.
the_fonz 8y ago
Regardless of how bad or good the 737 MAX was conceived, none of that matters because of regulatory capture of the FAA by way of millions of dollars of bribes paid to hundreds in Congress. Ever since the 737 NG, as the 2010 Al Jazeera report shows, Boeing has been installing substandard structural components and covering it up. No one should ever fly on an 737 NG or MAX aircraft because the regulatory environment was too weak, and so these aircraft are systematically-subpar. Defective structural parts and lax standards cannot be fixed with software upgrade. They're irreparably unsafe. Refusing to fly on them is the main logical act a passenger can undertake, and join Flyers' Rights http://flyersrights.org/ http://flyersrights.org/
- dehrmann 8y ago
  I'm on the fence on how bad the regulatory system is. Boeing has a massive incentive to make safe aircraft (they're seeing it first-hand right now), and modern commercial aircraft might be too complex for the FAA to effectively certify without input from manufacturers.
  - salawat 8y ago
    I vehemently disagree on the too complex front. If there is any inability to X going on in the FAA it is because of a culture of continual budget cuts hamstringing the ability of the agency to acquire and keep talent. The current incarnation of the FAA is a joke. This is not some bureaucratic paper pusher organization. This is the agency responsible for the overall architecture and requirements that historically made the American Aerospace industry a force to be reckoned with when they weren't being blindsided or hamstrung by excessive political pressure by industry. My proof? There is nothing complicated, about the trim system of a 737 MAX. I've sat down and done a bit of research, reverse engineered the basic mechanical components of the system, and I'm just some bloke who had a few hours to kill on the weekend. It's a highly geared mechanical gear train, with a secondary set of electrical motors, connected to a jackscrew, which drives the horizontal stabilizers while feeding back electrical actuation information back to the pilot's via the mechanical gear train. At some point, they wired in extra electrical connections from whatever black box MCAS is implemented in into the command circuit for the electric trim motor, and called it a day. There is nothing complicated there. It's a basic composition of simple machines. The complicated part is the implementation and logistics details, in addition to making sure at some point, all those pesky details are written down, all those integrated systems are tested/verified to be of sufficient quality, and that each component has an audit trail to be referenced should investigation be necessary. All that extra information is required specifically because it has been found to lead to increased safety over time. The FAA's duty is to ensure at a minimum, every player is living up to and respecting that standard. Something the history of budget cuts and delegation of authority to manufacturers has very clearly jeopardized. I don't like the FAA. I'll never even be able to take the physical pilot training to fly because of the draconian nature of their medical certification regulations, but I can't even argue that they are unjustified in terms of the catastrophic outcomes that are prevented by having them there. It is possible to hold a manufacturer to standards via external impartial audit. It is difficult to do so if you can't even support the billets for a set of Subject Matter Experts who don't need to resort to alternative revenue streams so they can concentrate on doing one thing and doing it well. There just needs to be the realization that at some point there must be a figure in the relationship between operator and manufacturer with the capacity, nay, responsibility to stop something which has not met the minimum standards of airworthyness. It ain't physics that's the problem, it's the people. Machines don't lie. Even when malfunctioning, they're telling the truth in as much as the physical principles they are subject to will allow.
tibbydudeza 8y ago
The Windows XP of passenger jets.
- bcatanzaro 8y ago
  Windows XP was actually pretty modern for its time. Maybe you mean Windows ME. ;)
  - Gibbon1 8y ago
    Thank you for reminding me of something I'd rather not remember. One thing I remember is Microsoft tried repeatedly with Win95, 98 and ME to get USB support to work correctly. Failing each time and having to start over. Much to the horror of peripheral manufacturers. With Windows XP that finally mostly worked.
filereaper 8y ago
I guess pilot retraining will always be time consuming, but can anything be done about recertification so that new designs don't face this issue?
- ajross 8y ago
  Good grief. No! Because as we are literally discovering, not knowing the details of how and why the automatic stabilization software is driving your trim wheel can kill you. The biggest single part of the failure chain here was the fact that this should have been an invasive change requiring recertification as a different aircraft and retraining for pilots instead of pretending that it was just like a 737-600.
  - Gibbon1 8y ago
    You are on my page. Type certification is the process to discover issues like this. I'm not familiar with pilot training and certification but it seems to me that even if the 737 MAX was type certified certifying a pilot with previous 737 experience would have been just supplemental training. Ultimately a nothing burger. I've seen a lot of cases where people get bent thinking of all the bad things that will happen because they think customers won't like some change. Always in the end if it's not a show stopper the customers just lump it.
CivBase 8y ago
Why exactly does this author think an airplane needs a lot of computing power? Airplane software does not deal with especially large amounts of data or intense computations. Faster hardware does not provide an inherent safety benefit. There is a large degree of coupling between avionics hardware and software amd updating that software is prohibitively expensive. There's just no point in doing it unless you get a tangible benefit. Furthermore, avionics systems take a long time to develop are not often rebuilt from scratch, so the hardware naturally lags far behind the computing power we're used to. The recycling of avionics systems is definitely fueled by a desire to reduce costs. However, it's also important to note that older, mature systems with a history of service inherently offer a degree of safety compared to something which is untested. The issue this article complains about can easily be seen as a safety feature. To be fair, the author does spend most of the article talking about missing features and design issues rather than harping on "processing power". It is a poor title.
- dmix 8y ago
  Wait until they finds out what software powers the nuclear weapons of 90%+ of the world, top-tier fighter jets, "critical" infrastructure like power (including nuclear), and countless financial/banking systems. To be fair the article doesn't critique using 1990s (or earlier) hardware/software. It's merely a clickbait headline. The only reference is dismissed by the very next sentence as insignificant [Boeing could be replaced with any ~~expert~~ somewhat-knowledgable person on the subject]: > The flight-control computers have roughly the processing power of 1990s home computers. A Boeing spokesman said the aircraft was designed with an appropriate level of technology to ensure safety. This only adds to my lowered standards when approaching NYT articles over the years. They've long been abandoning being correct for being entertaining at a seemingly accelerating rate. Which is too bad considering plenty of people continue to hold them to a higher credibility over the typical clickbaiter sites. Considering they are a subscription service I'm not convinced this is justified.
  - CivBase 8y ago
    > To be fair the article doesn't critique using 1990s (or earlier) hardware/software. I actually edited in an extra paragraph at the end of my original post because I wanted to at least credit the author for talking more about design issues than "processing power" in the body of the article. The title is very bad, though.
iliketosleep 8y ago
When it comes to flight computers it makes sense to be conservative when it comes to computing power - the goal should be simple, proven, and functional. But when it comes to diagnostics why not use more computing power? As the article states A second electronic system found on other Boeing jets also alerts pilots to unusual or hazardous situations during flight and lays out recommended steps to resolve them. Seems better than flipping through a paper manual when you've only a minute or two to save the plane.
- theclaw 8y ago
  Because new diagnostics mean new pilot training, meaning the new plane is not economical to buy when compared to the competition, meaning no plane at all. Those other Boeing jets required costly new training. Boeing were trying to build an update to the 737 that did not require this.
myth2018 8y ago
Mass media in general is stressing too much the fact that 737 MAX is allegedly based on dated designs. I think it's fair to say that MAX versions present some compromise solutions (like the now-infamous MCAS, which is there to compensate for the "unnatural" bigger engines). But I think that is not the main point. They would be good solutions it they worked as intended. There are some other more fundamental and more daunting, afaik unanswered questions. Like, why does such a critical system like MCAS take only a single AoA sensor as input, when there are two sensors available? Specially considering that the inputs from both are hardware-available to MCAS (the new software version is going to take data from both). Boeing affirmed in its manuals that the elevators would be able to compensate for the trimmed vertical stabilizers. Now the preliminary report in the Ethiopian's crash shows that the pilots wheren't able to perform such compensation, even by pulling the control columns all the way back. Those and some other issues are much more critical.
- kacamak 8y ago
  >Like, why does such a critical system like MCAS take only a single AoA sensor as input The classic approach is to have three sensors, so in case one fails you can know which one. Having two only indicates something is wrong but is not useful on the fly.
  - flyinglizard 8y ago
    Three with a disagree algorithm is definitely what I’d expect out of such a critical system, but two with signal averaging would still be much better than just one.
    - cm2187 8y ago
      Or alternatively with two, and disabling MCAS if they disagree, seems a better solution than having one and having no way to tell if it is working (keeping in mind both can still fail simultaneously). Not an ideal solution but better.
      - 8y ago
spacedog 8y ago
>> “They wanted to A, save money and B, to minimize the certification and flight-test costs" This seems to be the root cause of the problem. Competition is not always good.
- reacweb 8y ago
  I think Boeing was in a wrong mindset: certification was seen as a burden instead of being seen as an help to build a better aircraft. But maybe it was not only the fault of Boeing. The certification process of FAA is perhaps not friendly enough toward big air-frame changes (like putting the wings slightly higher to make room for the bigger reactor). Here, the certification process has not identified that a failure of one AOA sensor was able to cause a difficult to avoid crash. Maybe we should allow the use of less reliable source of data (like GPS, ...) each time a sensor has a problem.
stunt 8y ago
Except rumors from media there is no clear explanation about what is going on inside Boeing. I just hope Boeing management is not the old Microsoft management.
tzakrajs 8y ago
Three angle of attack sensors should have never been something that could be overridden by cost savings measures. The engineers wanted three sensors and the management said no without ensuring all planes sold had sufficient mitigating controls. We need justice.
buserror100 8y ago
I'd rather take a well tested i386 than a new processor that crashes Linux with microcode errors. Paper manuals actually instill confidence. Should pilots contact Stackoverflow in an emergency?
lqet 8y ago
Hm. I hesitate to defend Boeing here, but I think the outset of this article is a bit unfair. > Pilots start some new Boeing planes by turning a knob and flipping two switches. > The Boeing 737 Max, the newest passenger jet on the market, works differently. Pilots follow roughly the same seven steps used on the first 737 nearly 52 years ago: Shut off the cabin’s air-conditioning, redirect the air flow, switch on the engine, start the flow of fuel, revert the air flow, turn back on the air conditioning, and turn on a generator. So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it? > The strategy, to keep updating the plane rather than starting from scratch, offered competitive advantages. Pilots were comfortable flying it, while airlines didn’t have to invest in costly new training for their pilots and mechanics. For Boeing, it was also faster and cheaper to redesign and recertify than starting anew. > But the strategy has now left the company in crisis, following two deadly crashes in less than five months. How was it the strategy to keep updating the plane that left to this crisis? The strategy itself is not to blame here, and I very much like the idea to gradually improve a proven model. It was a bad execution of this strategy that left the company in crisis. In Germany, the national train agency Deutsche Bahn (and its predecessors) basically had a policy for nearly a century to order rolling stock that was designed to be produced for around 40 years. During this production run, the model was gradually improved. Some of the rolling stock designed in the 50ies is still in use, and quite reliable at that [0]. During the 90ies, agency and industry switched to a policy where basically every train generation was newly developed from scratch (for example, the ICE high speed trains). Guess what - you can channel a lot of public money into private hands that way, but bleeding edge technology is not what you want or need when you are trying to build a reliable transportation system. [0] https://de.wikipedia.org/wiki/N-Wagen https://de.wikipedia.org/wiki/N-Wagen
- throw0101a 8y ago
  > So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it? When you're nose down heading into the ground, taking your hand off the stick to adjust the trim can get panic-y. Especially since the MCAS may be working against your trim adjustments. The design of the Max line seems to be just enough compete with the A320neo, and just enough to not "need" re-certification. But these two justs, together, may have cost several hundred people their lives.
andy_ppp 8y ago
Does anyone else here feel aeroplane software should be mandated as being open source? Boeing would have never even dreamed of releasing the 737-Max software in the state it was in and the more eyes you have on things like this the better.
- StreamBright 8y ago
  Do you think there are a massive amount of developers out there who are at the same time avionics experts? I just don't see how being opensource would help. You wouldn't be able to even test these software projects because it requires and emulator or actual hardware to run on that you do not have. What would it chage if it was opensource?
  - andy_ppp 8y ago
    I think the argument you're making is completely false. First Premise is that there re not many avionics expers - I agree so having more of those experts able to look at review and learn from different implementations would be a good thing. Second premise is that you need to be an avionics expert to review and improve avionics software, I'm not convinced, there are many clever people out there. Third premise you can't test or run the software becuase there is no emulator; maybe if the software was open source someone would start writing an emulator - maybe an emulator could be open sourced? Fourth point; nothing would change if it was open source - I think the quality of code that would be released would go up and that's the main thing I'm talking about, embarrassment would be the optimal solution to something like this - I'd be very suprised if Boeing would release software that used a single sensor as input to a critical system like this.
    - StreamBright 8y ago
      I see so basically it would mean no benefits at all. Thanks for confirming. Now the other problem that I have this in this thread. The Boeing issue is not a software issue at all. Even if we chose the best case scenario for the software it would be still a physical design & lack of pilot education problem.
adontz 8y ago
While i386 may be enough to run a plane, what other software or hardware of the same age is still in use?
deleted 8y ago
[deleted]
throw0101a 8y ago
I'm sure this conversation will be looked at during discovery in any future lawsuits: > Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right. > Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes.
foldr 8y ago
>In the recent crashes, investigators believe the MCAS malfunctioned and moved a tail flap called the stabilizer, Fact checking at the NYT seems to be dead. The MCAS moves the entire stabilizer -- which is not a "flap" in any sense. Can they not find a pilot or some other knowledgeable person to read through an article like this before publishing it?

118 points by bushido 8y ago

The "1990s Computing Power" comment is ignorantly harsh. The MAX (to the best of my knowledge) uses a similar AMD29050-based architecture as the 777 (where it was pioneered) and the 737-800/900. It's amply fast and very very reliable, hardware and software (setting MCAS aside which is new to MAX). Any sane person would prefer the solid well-proven choice over the bleeding edge. People used to make similar comments about the AGC and the Space Shuttle GPCs e.g. that such-and-such-a-PC-was-faster. Yeah, so what?

londons_explore 8y ago
I'm not sure I agree here. There exist possibilities for far more safety with more compute. For example, when a mechanical failure occurs (for example an engine explodes and partially falls off) the flight characteristics of the plane change dramatically. Current systems fall back to human control, and hope that human will be able to figure out how to control a now very "unique" machine. Future systems could dynamically create new flight models based on collected data to be able to fly the machine in entirely new ways. For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? [1]. That sort of 'fly it how it has never been done before' is out of reach of human control, and might save lives. [1]:. https://www.ethz.ch/en/news-and-events/eth-news/news/2013/12/new-algorithm-makes-quadrocopters-safer.html https://www.ethz.ch/en/news-and-events/eth-news/news/2013/12...
- exelius 8y ago
  This tech is just now making it into military aircraft in the F-35 (it’s designed to compensate for failures or combat damage while keeping the plane’s handling relatively consistent), so I would guess it’ll be a while before we see a civilian version.
  - AnimalMuppet 8y ago
    "Figure out a way to fly the thing in the new, crippled configuration" is hard. Figure out a way that's better than what the human pilot came up with may be even harder.
    - londons_explore 8y ago
      It's hard, but it's a very measurable problem. You randomly deform the airframe in a physics simulator, then get a real pilot to try to fly the simulated damaged airframe to the nearest airport. You then get an AI to try doing the same. As soon as the AI manages to get to the airport more often than the human, the tech will save lives, on average. My guess is a PhD student could come up with an AI passing the above test inside a year. Yet aviation standards are stringent enough AI will be decades away from production use, if ever.
      - TheOtherHobbes 8y ago
        The political problem is that people still trust humans a lot more than they trust machines. So passengers would be nervous about being flown by an AI, even if the AI was safer. There's some weird psychology behind this: human pilots/drivers/etc are seen by passengers as a personal proxy, with agency over any situation. If you take away the proxy you take away the illusion of agency. Humans really do not like being put into situations where they believe they have no agency at all. As AI gets smarter, this will become more and more of a problem, until there some kind of cultural shift because AI is obviously safer it's not even a question any more. This may or may not happen.
      - notahacker 8y ago
        Your link to ZTE trumpeting the success of an algorithm helping an actual physical model of a quadcopter with a disabled rotor operating in lab conditions to crashland a bit more softly might be interpreted as a hint that the general case of safely landing a stricken aircraft is the sort of problem that probably isn't solved by a single PhD in a year... Keeping those with enough hubris to say "yeah, shouldn't take long if we've got some sort of sort of simulator to devise scenarios for a neural network to overfit to" about a wide range of failure modes that an entire field has spent decades studying well away from safety critical systems might be one of the more underappreciated aspects of tight regulation.
- jacquesm 8y ago
  > For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? The linked article does not support your point, it shows a quadcopter losing one rotor which then goes into a kind of 'controlled fall', two (or even three) rotors is never mentioned and even the one rotor missing scenario involves the whole craft spinning (see video on the page).
- theclaw 8y ago
  There also exists possibilities for far more danger. As you increase compute, you can run more code, and build more complicated interdependent components that are more difficult to test, especially for the sort of unexpected scenarios that "dynamic" behaviour would warrant. You would basically have a whole new plane - the opposite of what the 737 MAX 8 was intended to be.
wvenable 8y ago
> The "1990s Computing Power" comment is ignorantly harsh. I don't think so. You build a plane that's expected to run for decades -- if you put a 2010 computer in there now, it'll be 20 years old in 2030. That 1990's computer was put there in the 90's, does that really make it more reliable?
- pinewurst 8y ago
  That so-called 1990s computer is probably a year old, embedded in a Honeywell module that’ll be available new for the next 15 years.
  - wvenable 8y ago
    Of course. It's not literally a 1990's computer. But the idea that it's just somehow superior because the design is 3 decades old doesn't make any sense. 30 years ago it would have been new technology; so where was that same argument then?
    - deleted 8y ago
      [deleted]
    - bronson 8y ago
      It's had 3 decades of experience and bugfixes.
      - wvenable 8y ago
        If new developing is happening, as it was in this case, that doesn't really matter. This problem was amplified by maintaining old technology, not mitigated by it.
        pinewurst 8y ago
        I don’t agree. The failure(s) weren’t in the old stuff, they were in the poor and ultimately unsafe cover-up (for lack of a better word) of the new. The CG change from the new engines specifically, being masked by new software, MCAS. The only old part culpable is the legacy 737 limit of 2 AoA sensors (with only 1 used for MCAS input (a new thing)).
BuildTheRobots 8y ago
The "1990's Computer Power" comment is factually correct. According to wikipedia, the AMD29000 processor was first released in 1988 and "In late 1995 AMD dropped development of the 29k..." I'm not commenting on whether that's appropriate to be running your aircraft or not, but it does very much seem to be a 1990's CPU running approx 50mhz.

chmaynard 8y ago

The entire management chain that dictated these design compromises should be surgically removed from Boeing and prosecuted for negligence. Of course, that will never happen.

PhantomGremlin 8y ago
This doesn't get said enough. Within days of the Lion Air crash, MCAS was strongly suspected to be the problem. And yet Boeing management allowed 157 more people to die in the Ethiopian Airlines crash. And then the Boeing CEO still tried to tell everyone that it was a safe airplane. The entire top management of Boeing, and the entire board of directors, should be perp-walked out the door. Of course, that will never happen.

kbos87 8y ago

The thing that baffles me - why the hell has there never been any innovation in the form of making it easier to diagnose a problem and move through the necessary steps in a checklist beyond fumbling through a paper manual? In a situation where seconds count, it seems illogical that this has never been improved upon. Didn’t at least one of these planes crash because the pilots are believed to have only gotten X number of steps through a list of possible reasons?

InTheArena 8y ago
The entire aviation business and safety structure is build on extraordinary rigidity to processes and approaches. In addition, automation is viewed as a threat to multiple worker groups. Sometimes this is justified -certainly the MCAS incident is a example. Changes like the MCAS should never have been allowed to change a fundamental part of the airplane interface without considerably more processes being met. That said, sometimes opposition to enhancements is trade unionists being trade unionists.
Qqqwxs 8y ago
I remember hearing about the US Airways flight that had an emergency landing in the Hudson river (the film 'Sully' is based on this). The plane sunk faster than it should have because the checklist item 'close the vents' was too far down to be completed before the water landing. In response, the checklist was edited and the item was moved further to the top. Apologies for not finding an exact link, but here is the wikipedia article for the flight: https://www.wikiwand.com/en/US_Airways_Flight_1549 https://www.wikiwand.com/en/US_Airways_Flight_1549
bdonlan 8y ago
The newer airbuses actually do have electronic checklists including - for at least some steps - automatic detection that the step has been completed. You can see an example of the electronic checklist in this simulator video: https://www.youtube.com/watch?v=jzjyr0UZKRw https://www.youtube.com/watch?v=jzjyr0UZKRw Even on aircraft without fully integrated checklists, many operators have switched to using iPads instead of paper manuals as well.

whoisjuan 8y ago

Kind of unfair. Do they really think that an aircraft should be powered by the same software or hardware that powers a modern computer or smartphone? An aircraft needs incredible reliability. You don’t get that with modern software/hardware architectures.

da_chicken 8y ago
Yeah, the B-52, adopted in 1952, is planned to be is service through the 2050s. An airframe design expected to last for over 100 years. Keep in mind that flight itself is barely 100 years old. And paper manuals? Are they expecting to use iPads for in flight documents or something? Shall we compare failure states of paper vs tablets?
- TylerE 8y ago
  There are still DC-3s from the early '30s in service.
  - Aloha 8y ago
    The old saying is "the only replacement for a DC-3 is another DC-3" Though I think most of them that are flying are C-47's from the war era.
- tedivm 8y ago
  That article says that other plans will display the error and the recommended checklist automatically, which certainly seems faster than flipping through a bunch of books trying to find a specific page based off of some lights.
  - ReptileMan 8y ago
    A lecturer of mine once said - knowing what kind of people we teach here for programmers, I am scared of going to the doctor. The less software is in something(and less cloudy/iot) the better. And the more I program the more I prefer stupid and mechanical things when reliability is on the line. A paper manual is always there for you.

the_fonz 8y ago

Regardless of how bad or good the 737 MAX was conceived, none of that matters because of regulatory capture of the FAA by way of millions of dollars of bribes paid to hundreds in Congress. Ever since the 737 NG, as the 2010 Al Jazeera report shows, Boeing has been installing substandard structural components and covering it up. No one should ever fly on an 737 NG or MAX aircraft because the regulatory environment was too weak, and so these aircraft are systematically-subpar. Defective structural parts and lax standards cannot be fixed with software upgrade. They're irreparably unsafe. Refusing to fly on them is the main logical act a passenger can undertake, and join Flyers' Rights http://flyersrights.org/ http://flyersrights.org/

dehrmann 8y ago
I'm on the fence on how bad the regulatory system is. Boeing has a massive incentive to make safe aircraft (they're seeing it first-hand right now), and modern commercial aircraft might be too complex for the FAA to effectively certify without input from manufacturers.
- salawat 8y ago
  I vehemently disagree on the too complex front. If there is any inability to X going on in the FAA it is because of a culture of continual budget cuts hamstringing the ability of the agency to acquire and keep talent. The current incarnation of the FAA is a joke. This is not some bureaucratic paper pusher organization. This is the agency responsible for the overall architecture and requirements that historically made the American Aerospace industry a force to be reckoned with when they weren't being blindsided or hamstrung by excessive political pressure by industry. My proof? There is nothing complicated, about the trim system of a 737 MAX. I've sat down and done a bit of research, reverse engineered the basic mechanical components of the system, and I'm just some bloke who had a few hours to kill on the weekend. It's a highly geared mechanical gear train, with a secondary set of electrical motors, connected to a jackscrew, which drives the horizontal stabilizers while feeding back electrical actuation information back to the pilot's via the mechanical gear train. At some point, they wired in extra electrical connections from whatever black box MCAS is implemented in into the command circuit for the electric trim motor, and called it a day. There is nothing complicated there. It's a basic composition of simple machines. The complicated part is the implementation and logistics details, in addition to making sure at some point, all those pesky details are written down, all those integrated systems are tested/verified to be of sufficient quality, and that each component has an audit trail to be referenced should investigation be necessary. All that extra information is required specifically because it has been found to lead to increased safety over time. The FAA's duty is to ensure at a minimum, every player is living up to and respecting that standard. Something the history of budget cuts and delegation of authority to manufacturers has very clearly jeopardized. I don't like the FAA. I'll never even be able to take the physical pilot training to fly because of the draconian nature of their medical certification regulations, but I can't even argue that they are unjustified in terms of the catastrophic outcomes that are prevented by having them there. It is possible to hold a manufacturer to standards via external impartial audit. It is difficult to do so if you can't even support the billets for a set of Subject Matter Experts who don't need to resort to alternative revenue streams so they can concentrate on doing one thing and doing it well. There just needs to be the realization that at some point there must be a figure in the relationship between operator and manufacturer with the capacity, nay, responsibility to stop something which has not met the minimum standards of airworthyness. It ain't physics that's the problem, it's the people. Machines don't lie. Even when malfunctioning, they're telling the truth in as much as the physical principles they are subject to will allow.

tibbydudeza 8y ago

The Windows XP of passenger jets.

bcatanzaro 8y ago
Windows XP was actually pretty modern for its time. Maybe you mean Windows ME. ;)
- Gibbon1 8y ago
  Thank you for reminding me of something I'd rather not remember. One thing I remember is Microsoft tried repeatedly with Win95, 98 and ME to get USB support to work correctly. Failing each time and having to start over. Much to the horror of peripheral manufacturers. With Windows XP that finally mostly worked.

filereaper 8y ago

I guess pilot retraining will always be time consuming, but can anything be done about recertification so that new designs don't face this issue?

ajross 8y ago
Good grief. No! Because as we are literally discovering, not knowing the details of how and why the automatic stabilization software is driving your trim wheel can kill you. The biggest single part of the failure chain here was the fact that this should have been an invasive change requiring recertification as a different aircraft and retraining for pilots instead of pretending that it was just like a 737-600.
- Gibbon1 8y ago
  You are on my page. Type certification is the process to discover issues like this. I'm not familiar with pilot training and certification but it seems to me that even if the 737 MAX was type certified certifying a pilot with previous 737 experience would have been just supplemental training. Ultimately a nothing burger. I've seen a lot of cases where people get bent thinking of all the bad things that will happen because they think customers won't like some change. Always in the end if it's not a show stopper the customers just lump it.

CivBase 8y ago

Why exactly does this author think an airplane needs a lot of computing power? Airplane software does not deal with especially large amounts of data or intense computations. Faster hardware does not provide an inherent safety benefit. There is a large degree of coupling between avionics hardware and software amd updating that software is prohibitively expensive. There's just no point in doing it unless you get a tangible benefit. Furthermore, avionics systems take a long time to develop are not often rebuilt from scratch, so the hardware naturally lags far behind the computing power we're used to. The recycling of avionics systems is definitely fueled by a desire to reduce costs. However, it's also important to note that older, mature systems with a history of service inherently offer a degree of safety compared to something which is untested. The issue this article complains about can easily be seen as a safety feature. To be fair, the author does spend most of the article talking about missing features and design issues rather than harping on "processing power". It is a poor title.

dmix 8y ago
Wait until they finds out what software powers the nuclear weapons of 90%+ of the world, top-tier fighter jets, "critical" infrastructure like power (including nuclear), and countless financial/banking systems. To be fair the article doesn't critique using 1990s (or earlier) hardware/software. It's merely a clickbait headline. The only reference is dismissed by the very next sentence as insignificant [Boeing could be replaced with any ~~expert~~ somewhat-knowledgable person on the subject]: > The flight-control computers have roughly the processing power of 1990s home computers. A Boeing spokesman said the aircraft was designed with an appropriate level of technology to ensure safety. This only adds to my lowered standards when approaching NYT articles over the years. They've long been abandoning being correct for being entertaining at a seemingly accelerating rate. Which is too bad considering plenty of people continue to hold them to a higher credibility over the typical clickbaiter sites. Considering they are a subscription service I'm not convinced this is justified.
- CivBase 8y ago
  > To be fair the article doesn't critique using 1990s (or earlier) hardware/software. I actually edited in an extra paragraph at the end of my original post because I wanted to at least credit the author for talking more about design issues than "processing power" in the body of the article. The title is very bad, though.

iliketosleep 8y ago

When it comes to flight computers it makes sense to be conservative when it comes to computing power - the goal should be simple, proven, and functional. But when it comes to diagnostics why not use more computing power? As the article states A second electronic system found on other Boeing jets also alerts pilots to unusual or hazardous situations during flight and lays out recommended steps to resolve them. Seems better than flipping through a paper manual when you've only a minute or two to save the plane.

theclaw 8y ago
Because new diagnostics mean new pilot training, meaning the new plane is not economical to buy when compared to the competition, meaning no plane at all. Those other Boeing jets required costly new training. Boeing were trying to build an update to the 737 that did not require this.

myth2018 8y ago

Mass media in general is stressing too much the fact that 737 MAX is allegedly based on dated designs. I think it's fair to say that MAX versions present some compromise solutions (like the now-infamous MCAS, which is there to compensate for the "unnatural" bigger engines). But I think that is not the main point. They would be good solutions it they worked as intended. There are some other more fundamental and more daunting, afaik unanswered questions. Like, why does such a critical system like MCAS take only a single AoA sensor as input, when there are two sensors available? Specially considering that the inputs from both are hardware-available to MCAS (the new software version is going to take data from both). Boeing affirmed in its manuals that the elevators would be able to compensate for the trimmed vertical stabilizers. Now the preliminary report in the Ethiopian's crash shows that the pilots wheren't able to perform such compensation, even by pulling the control columns all the way back. Those and some other issues are much more critical.

kacamak 8y ago
>Like, why does such a critical system like MCAS take only a single AoA sensor as input The classic approach is to have three sensors, so in case one fails you can know which one. Having two only indicates something is wrong but is not useful on the fly.
- flyinglizard 8y ago
  Three with a disagree algorithm is definitely what I’d expect out of such a critical system, but two with signal averaging would still be much better than just one.
  - cm2187 8y ago
    Or alternatively with two, and disabling MCAS if they disagree, seems a better solution than having one and having no way to tell if it is working (keeping in mind both can still fail simultaneously). Not an ideal solution but better.
    - 8y ago

spacedog 8y ago

>> “They wanted to A, save money and B, to minimize the certification and flight-test costs" This seems to be the root cause of the problem. Competition is not always good.

reacweb 8y ago
I think Boeing was in a wrong mindset: certification was seen as a burden instead of being seen as an help to build a better aircraft. But maybe it was not only the fault of Boeing. The certification process of FAA is perhaps not friendly enough toward big air-frame changes (like putting the wings slightly higher to make room for the bigger reactor). Here, the certification process has not identified that a failure of one AOA sensor was able to cause a difficult to avoid crash. Maybe we should allow the use of less reliable source of data (like GPS, ...) each time a sensor has a problem.

stunt 8y ago

Except rumors from media there is no clear explanation about what is going on inside Boeing. I just hope Boeing management is not the old Microsoft management.

tzakrajs 8y ago

Three angle of attack sensors should have never been something that could be overridden by cost savings measures. The engineers wanted three sensors and the management said no without ensuring all planes sold had sufficient mitigating controls. We need justice.

buserror100 8y ago

I'd rather take a well tested i386 than a new processor that crashes Linux with microcode errors. Paper manuals actually instill confidence. Should pilots contact Stackoverflow in an emergency?

lqet 8y ago

Hm. I hesitate to defend Boeing here, but I think the outset of this article is a bit unfair. > Pilots start some new Boeing planes by turning a knob and flipping two switches. > The Boeing 737 Max, the newest passenger jet on the market, works differently. Pilots follow roughly the same seven steps used on the first 737 nearly 52 years ago: Shut off the cabin’s air-conditioning, redirect the air flow, switch on the engine, start the flow of fuel, revert the air flow, turn back on the air conditioning, and turn on a generator. So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it? > The strategy, to keep updating the plane rather than starting from scratch, offered competitive advantages. Pilots were comfortable flying it, while airlines didn’t have to invest in costly new training for their pilots and mechanics. For Boeing, it was also faster and cheaper to redesign and recertify than starting anew. > But the strategy has now left the company in crisis, following two deadly crashes in less than five months. How was it the strategy to keep updating the plane that left to this crisis? The strategy itself is not to blame here, and I very much like the idea to gradually improve a proven model. It was a bad execution of this strategy that left the company in crisis. In Germany, the national train agency Deutsche Bahn (and its predecessors) basically had a policy for nearly a century to order rolling stock that was designed to be produced for around 40 years. During this production run, the model was gradually improved. Some of the rolling stock designed in the 50ies is still in use, and quite reliable at that [0]. During the 90ies, agency and industry switched to a policy where basically every train generation was newly developed from scratch (for example, the ICE high speed trains). Guess what - you can channel a lot of public money into private hands that way, but bleeding edge technology is not what you want or need when you are trying to build a reliable transportation system. [0] https://de.wikipedia.org/wiki/N-Wagen https://de.wikipedia.org/wiki/N-Wagen

throw0101a 8y ago
> So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it? When you're nose down heading into the ground, taking your hand off the stick to adjust the trim can get panic-y. Especially since the MCAS may be working against your trim adjustments. The design of the Max line seems to be just enough compete with the A320neo, and just enough to not "need" re-certification. But these two justs, together, may have cost several hundred people their lives.

andy_ppp 8y ago

Does anyone else here feel aeroplane software should be mandated as being open source? Boeing would have never even dreamed of releasing the 737-Max software in the state it was in and the more eyes you have on things like this the better.

StreamBright 8y ago
Do you think there are a massive amount of developers out there who are at the same time avionics experts? I just don't see how being opensource would help. You wouldn't be able to even test these software projects because it requires and emulator or actual hardware to run on that you do not have. What would it chage if it was opensource?
- andy_ppp 8y ago
  I think the argument you're making is completely false. First Premise is that there re not many avionics expers - I agree so having more of those experts able to look at review and learn from different implementations would be a good thing. Second premise is that you need to be an avionics expert to review and improve avionics software, I'm not convinced, there are many clever people out there. Third premise you can't test or run the software becuase there is no emulator; maybe if the software was open source someone would start writing an emulator - maybe an emulator could be open sourced? Fourth point; nothing would change if it was open source - I think the quality of code that would be released would go up and that's the main thing I'm talking about, embarrassment would be the optimal solution to something like this - I'd be very suprised if Boeing would release software that used a single sensor as input to a critical system like this.
  - StreamBright 8y ago
    I see so basically it would mean no benefits at all. Thanks for confirming. Now the other problem that I have this in this thread. The Boeing issue is not a software issue at all. Even if we chose the best case scenario for the software it would be still a physical design & lack of pilot education problem.

adontz 8y ago

While i386 may be enough to run a plane, what other software or hardware of the same age is still in use?

deleted 8y ago

[deleted]

throw0101a 8y ago

I'm sure this conversation will be looked at during discovery in any future lawsuits: > Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right. > Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes.

foldr 8y ago

>In the recent crashes, investigators believe the MCAS malfunctioned and moved a tail flap called the stabilizer, Fact checking at the NYT seems to be dead. The MCAS moves the entire stabilizer -- which is not a "flap" in any sense. Can they not find a pilot or some other knowledgeable person to read through an article like this before publishing it?