25 ms·
Boeing software under scrutiny as Ethiopia prepares crash report
- ceejayoz 8y agoThat answers the "why didn't they remember the airworthiness notice" question - they did, and tried to disable it, and it re-enabled itself. Glad I didn't pick up Boeing shares on the cheap just yet.
- mirashii 8y ago> it re-enabled itself No, the possibility that it reenabled itself is being investigated, but there's no reason to conclude that yet.
- danaliv 8y agoAmazing how easily “the possibility of X is being investigated” turns into “X happened.”
- dsfyu404ed 8y agoWhen there's a stake at the ready the bar for being a witch becomes lower. People form a version of events in their head and then cling to it for dear life. This whole thing should also serve as a reminder why we don't rely on internet mobs to investigate important things.
- ceejayoz 8y ago> This whole thing should also serve as a reminder why we don't rely on internet mobs to investigate important things. I'm more inclined to take it as a warning on regulatory capture, especially after it took most of the rest of the world grounding the craft to get the FAA to do so.
- everdev 8y ago> Glad I didn't pick up Boeing shares on the cheap just yet. Unfortunately, it's a focus on profits over people that likely led to these disasters. I don't know how you can stop or dissuade from people making a buck off of a tragedy, but it might be worth putting our attention on the human element of this mess. Something about merging this safety issue with an opportunity to "buy low" and make some cash just doesn't sit right.
- dboreham 8y agoTraditionally this is done by having a separate safety, QA or compliance organization (depending on industry) that has the power to overrule the bean counters and guys with white teeth.
- bilbo0s 8y agoThat's how it works in the FDA regulated medical device and medical software industry. We had a woman who could shut us down on a whim. (And she would, from time to time.) That said, in the FDA regulated industry, every change was signed by not only the engineer, but by the QA person, by the engineering manager, and by her. So in fairness to her, SHE would have been one of the people going to jail if the software told a doctor to take out the wrong kidney because of a bug in the calculation of an ortho normal basis or something. So I didn't blame her one bit.
- inferiorhuman 8y agoGlad I didn't pick up Boeing shares on the cheap just yet. Look at a six month graph, Boeing is trending up and the stock price is back to where it was at the end of January 2019. Yeah, gross.
- CamperBob2 8y agoBoeing is the very definition of "too big to fail." They'll be fine.
- inferiorhuman 8y agoYeah, that's a bit of a problem isn't it?
- cjbprime 8y agoThat's not what the article says. The most likely situation from the article is the pilots: * disabled the trim motor * realized they were unable to trim back to level flight without it, probably due to aerodynamic load on the stabilizer * turned it back on * MCAS was still enabled -- it can't be disabled directly -- and used the trim motor itself to nose-down more, making the situation unrecoverable.
- Someone1234 8y agoI strongly suggest people skip this re-hosted article, and instead read the Reuters one here: Title: Boeing software under scrutiny as Ethiopia prepares crash report https://www.reuters.com/article/us-ethiopia-airplane-software/exclusive-boeing-software-engaged-repeatedly-before-crash-sources-idUSKCN1RF0YU https://www.reuters.com/article/us-ethiopia-airplane-softwar... I have numerous problems with Arstechnica's re-write, but suffice to say that nuance is lost and questionable conclusory statements added. I will say in the case of BOTH articles, it sounds like we need to wait and see. Neither article has an answer for how auto-trim was re-actived, be it the pilots or something else. The answer to that question may have wide reaching implications.
- sctb 8y agoThanks! We've updated the link from https://arstechnica.com/information-technology/2019/04/ethiopian-air-pilots-turned-off-737-max-anti-stall-system-then-it-turned-on-again/ https://arstechnica.com/information-technology/2019/04/ethio....
- digikata 8y agoThis other HN submission has some interesting hypotheticals on why the system might have been reactivated multiple times: https://news.ycombinator.com/item?id=19565056 https://news.ycombinator.com/item?id=19565056
- acqq 8y agoThis analysis sounds very plausible (all emphasis marks mine): https://leehamnews.com/2019/04/03/et302-used-the-cut-out-switches-to-stop-mcas/ https://leehamnews.com/2019/04/03/et302-used-the-cut-out-swi... "the high speeds observed in ET302’s FlightRadar24 trace (Figure 1) were logical. It’s a consequence of following the Emergency checklist for “IAS disagree” (IAS is Indicated Airspeed, i.e. the dynamic air pressure experienced by the aircraft) after takeoff. The combination of the preceding checklist followed by an MCAS Trim Runaway checklist could create a situation where manual trimming after a Trim Cut-Out would be difficult to impossible and would require non-checklist actions." "At a miss-trimmed Stabilator, you either have to re-engage Electric trim or off-load the Stabilator jackscrew by stick forward, creating a nose-down bunt maneuver, followed by trim. Stick forward to trim was not an option for ET302, they were at 1,000ft above ground. According to The Wall Street Journal, the ET302 crew re-engaged electrical trim to save the situation, to get the nose up. It was their only chance. But too late. The aggressive MCAS kicked in and worsened the situation before they could counter it." The Airport elevation of Addis Ababa airport is 7625 feet AMSL. The highest point of the flight recorded by the flightradar24 is 8600 ft AMSL which gives the plane's altitude of around only 1000 ft at highest! The speeds relative to the ground recorded near the end were 380 kt, i.e. 640 ft/s. Obviously every second mattered, and at the above speeds, the plane was only a few seconds from crash away when the MCAS was pointing the nose down.
- deleted 8y ago[deleted]
- pupppet 8y agoBoeing deserves to get nailed for this. You can't tell me they didn't know it was the MCAS at fault after the first crash 6 months ago, and decided to do...nothing.
- all_blue_chucks 8y agoFrom a legal standpoint, shipping a "fix" could be used as evidence of guilt, though.
- hangonhn 8y agoThings like that is why we have such insane damages dollar amounts for punitive damage. People reward huge punitive damages since the Ford Pinto case to compel companies to "do the right thing" because if you don't and you're found guilty, the punitive damage can be many times what the original cost would have been.
- jgust 8y agoI've been saying that software engineering as an industry is in the caveman state of development. The amount of scrutiny that production software gets at _most_ companies is hilariously lacking compared to say, a civil engineering blueprint and there are virtually zero repercussions for failure or negligence. Compare the education & certification required to be called a civil engineer vs. a software engineer and you'll notice that the industry is basically the wild west. You could also argue that computer science isn't yet a "hard science" like mechanical, civil, chemical, etc. engineering are. I predict that there will be a split in the industry between "software engineer" and "software laborer" where there are specific certifications you need to have before being called an "engineer". "Software engineers" will be held responsible, like civil engineers are, for mistakes that result in accidents. Companies that write safety critical software like auto, air, and space industries where human lives or huge $$$ are on the line will hire engineers, e-commerce companies will continue to hire "code laborers". Maybe they will even give you a fancy ring to wear?
- 8y ago
- CodeWriter23 8y agoNote to self: eventual consistency good for web/mobile apps, bad for avionics.
- ummonk 8y agoDepends on the app. There are a lot of web / mobile app use cases where stronger consistency is needed, e.g. banking.
- cjbprime 8y agoYou'd think, but banking (e.g. the ATM network) is all about choosing availability over consistency. If the ATM network can't reach your bank to discover your balance, it'll just let you withdraw and figure your bank will sort things out later after it comes back online.
- tzs 8y agoI just had a 737 MAX fly low almost over my house [1]. That descending left turn at around 2300 ft was centered on roughly my neighborhood. (The flight track shown on Flightware is displaced to the NW a bit). [1] https://flightaware.com/live/flight/BOE1/history/20190403/1700ZZ/KBFI/KBFI https://flightaware.com/live/flight/BOE1/history/20190403/17...
- yarosv 8y agoI thought they were grounded. Edit: Checked flightaware, it was Boing itself flying it.
- rconti 8y agoOnly from commercial service
- danaliv 8y agoSort of. All flights in U.S. territory, and by U.S. airlines anywhere in the world, are banned. Non-passenger flights can be authorized but they require either a Special Flight Permit (a specific process for authorizing ferry flights of any non-airworthy aircraft) or an Experimental Airworthiness Certificate (which you may know of from homebuilts, but in this case it's for testing design changes).[1] 1. https://www.faa.gov/news/updates/media/Emergency_Order.pdf https://www.faa.gov/news/updates/media/Emergency_Order.pdf
- msbarnett 8y agoBoeing's flying test-flights to test software fixes etc out of their Seattle facility
- donarb 8y agoSouthwest Airlines was allowed to fly all of their grounded jets to a facility in Arizona a few weeks after the initial grounding. The FAA allows this as long as there are no passengers on board.
- drawkbox 8y ago
- keyme 8y agoSo the physical cutoff switches are still just inputs to software? WTF.
- bilbo0s 8y agoIf that's true, it's inexcusable. But we should wait to find out more information.
- ams6110 8y agoNo, the working theory I've heard is that the hand-crank trim wheels may have been overwhelmed by the air load on the stabilizer given its "nose down" configuration, probable full "nose up" pull on the elevators, and the very high speed they were flying. Speculation is they may have turned the electric trim back on to use the manual trim switches to trim up. By that time possibly too late or even the electric trim was overwhelmed by the load, and of course this would have brought MCAS back into play also.
- bilbo0s 8y agoAgain, instead of relying on speculation, we should wait to get more information.
- ams6110 8y agoAgreed. Hard not to speculate though. I personally find this stuff fascinating from a technical and human factors perspective -- the deaths are tragic of course.
- msbarnett 8y agoThis isn't rumor mongering. This is straight out of Boeing's own documentation on the 737: https://www.satcom.guru/2019/04/stabilizer-trim-loads-and-range.html https://www.satcom.guru/2019/04/stabilizer-trim-loads-and-ra... When the stabilizer is extended and is being counteracted by elevator inputs, the forces on the control column make manually trimming stabilizers difficult or impossible. Boeing instructs that elevator input on the stick must be eased back in order to manually adjust the stabilizers.
- mlamat 8y agoYou can't fix a hardware problem with software.
- fuckyah 8y agowhen's the press conference?
- mindslight 8y agoPure handwaving speculation here, but this incident is feeling like a result of the limitations of fly-by-cable - specifically taking the tried and true design of fly-by-cable, but augmenting it with motors in parallel to human control. System integration is a known common source of errors, and what is less formally specified than the effective limits of a human being? If the human(s) are fully in the loop, then the buck stops with them and we know how to socially characterize and educate about failure modes. A pure-computer portion of the loop is characterizable by straight engineering. But rather, we've got human pilots physically fighting against electrical motors. Either the pilot is right and the computer is wrong (in which case any physical effort expended is a further detriment), or the computer is right and the pilot is wrong (in which case letting the pilot effectively sabotage the computer is wrong). There is no middle ground between these two - weighing the input of a human versus the input of a computer based on them physically fighting is not a sound methodology. Either powered drive of the trim is necessary or it is not. As it seems to be necessary due to the forces involved, the sensible design is to declare the actuator a critical part of the plane, and make the human input go through the actuator rather than around it.
- mdekkers 8y agoAvionics software that has direct impact on the handling of a passenger flight should be globally mandated to be guaranteed faultless. There should be direct liability to the software engineering team. I know Boeing's software wasn't written in the USA, and was outsourced to the lowest bidder that met some on-paper requirements. If you think this is outlandish, read "They Write The Right Stuff" https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff It can be done, but it is simply expensive. Boeing should hang for this.