19 ms·

Pilot Who Hitched a Ride Saved Lion Air 737 on the Day Before Deadly Crash

327 points by erict15 8y ago

samfisher83 8y ago
Suprised they don't have a lessons learned portal. Would have saved some lives.
- zkms 8y ago
  That's the sort of thing the NASA report system is for, though IDK the latencies involved in the system: https://en.wikipedia.org/wiki/Aviation_Safety_Reporting_System https://en.wikipedia.org/wiki/Aviation_Safety_Reporting_Syst... > The Aviation Safety Reporting System, or ASRS, is the US Federal Aviation Administration's (FAA) voluntary confidential reporting system that allows pilots and other aircraft crew members to confidentially report near misses and close calls in the interest of improving air safety.
- WrtCdEvrydy 8y ago
  Can you imagine if the public saw how often planes were close to disaster?
  - inamberclad 8y ago
    Don't confuse apathy with unavailability. The public (and it sounds like you as well) just doesn't look. Here's NASA's safety database: https://asrs.arc.nasa.gov/ https://asrs.arc.nasa.gov/
  - userbinator 8y ago
    They already can: http://avherald.com/ http://avherald.com/
- JMTQp8lwXL 8y ago
  Only as useful as people actually reading from such a portal.
nateabele 8y ago
> [T]hey got help from an unexpected source: an off-duty pilot who happened to be riding in the cockpit. That extra pilot, who was seated in the cockpit jumpseat [...] Am I missing something here? Isn't it normal for off-duty pilots to ride in the jump-seat?
- inamberclad 8y ago
  Yes. If the seat is unoccupied, airlines will usually let off duty pilots use it to get between different airports. If I'm remembering correctly, it's done for pilots from other airlines as well.
- khazhou 8y ago
  Sure. I don't think they're implying otherwise. The lucky coincidence is that he happened to be aware of the issue and the fix.
- latch 8y ago
  Well, you're missing the point of the story which has nothing to do with the frequency of off-duty pilots hitching a ride.
- koolba 8y ago
  Riding dead head as a pilot is normal. They do it all the time to get back home or wherever their next flight is from. Being in the right place, and happening to know exactly how to deal with what would otherwise kill the pilot and all the passengers, is incredibly fortunate.
  - ruytlm 8y ago
    I suppose similar in terms of the right place a the right time would be the QF32 incident[0], where by chance there were two additional pilots in the cockpit; a check captain, and a supervising check captain who was training that check captain. [0]: https://en.wikipedia.org/wiki/Qantas_Flight_32 https://en.wikipedia.org/wiki/Qantas_Flight_32
    - 8y ago
xvf22 8y ago
An extra set of eyes saved them, it's a shame that there wasn't any way for them to include a reminder in the next crews flight plans.
- sundvor 8y ago
  It is pretty shocking this wasn't noted as a serious incident needing investigation before more flights were undertaken.
jnbiche 8y ago
...
- jimktrains2 8y ago
  Do we know if the issue is bad software or bad specs? Even formal methods won't save you from a bad spec.
  - khuey 8y ago
    I don't think we've seen anything to indicate that the problem is not a bad spec.
    - jnbiche 8y ago
      We haven't seen anything either way. Even the allegation that it's the MCAS system is totally unconfirmed to the public.
  - jnbiche 8y ago
    Yes, and I mentioned this in my second paragraph. And no, we the public know very little about this point about the current alleged MCAS problem, but we know that many problems with safety critical software comes from implementations, not specs.
- platinumrad 8y ago
  >In fact, I thought these crash recovery systems had to be written by 3 different teams in 3 different languages or architectures, and only activate when 2 of the 3 signal true. Or there's some manufacturer that does avionic software like this, I forget who. Redundant systems with majority voting are common, and this can include multiple implementations of the same spec, but I'm not aware of any manufacturer that intentionally uses three different languages or architectures.
  - jimktrains2 8y ago
    I don't know about 3 languages, but https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff implies that there is software written by at least two different groups running on the shuttle. I've also heard anecdotally that some train signalling and on-board control systems run software simultaneously in linuc and freebsd to oessen the chance of simultaneous memory faults. (I didn't get a chance to ask for more details or clarification.)
reasonablemann 8y ago
Is it possible for Boeing engineers to lose their professional status as a result of this situation?
- cloakandswagger 8y ago
  Over pilot error and an unmaintained airplane? Doubtful.
  - _jal 8y ago
    You seem very certain of the cause(s) of the crash. Care to share the source of your knowledge?
    - rootusrootus 8y ago
      Just as certain as everyone else on HN that knows it's all Boeing's fault.
      - _jal 8y ago
        Partly my point. Plenty if internet experts to go around.
  - sokoloff 8y ago
    I agree that the pilots likely shoulder some of the blame (and in an NTSB-investigated crash, I'd expect their failure to follow the non-normal checklist memory items to be the primary cause), it's not enough to say that this was simple pilot error and poor maintenance. Boeing's going to wear some of the blame here, as is proper, IMO.
  - ceejayoz 8y ago
    From the article: "There have been no reports of maintenance issues with the Ethiopian Airlines plane before its crash."
- jimktrains2 8y ago
  Wouldn't a better point of view be: shouldn't we reëvaluate the items that allow self-certification and fund the FAA properly so that they can certify medium/large change themselves instead of letting the manufacturer? Self-certification has a place, but it should always be accompanied by random checks and shouldn't be for anything large, critical, or first time through. Edit: Better in that it helps solve what is largely a political and not an engineering problem.
pimlottc 8y ago
The headline is really confusing; I thought I was perhaps reading it wrong, but it's sort of impossible to read correctly until you realize they are talking about a separate incident than the well-known crash. It would have been clearer if they had written something simpler, like "Lion Air 737 Nearly Crashed One Day Before Deadly Accident"
- JMTQp8lwXL 8y ago
  Agreed. I ended up reading a second article on the topic to ensure I was understanding it correctly (the prior day's flight involving the same identical aircraft, not a different one of the same model).
- system2 8y ago
  I read it 5 times and I am glad I wasn't the one. I thought it was 737 days before.
- foobarbecue 8y ago
  I still can't tell if it says whether or was actually the same individual aircraft.
  - RyJones 8y ago
    Same aircraft
- dang 8y ago
  We added "on the" to try to make that clearer above.
jquery 8y ago
Runaway trim is supposed to be part of the "memory checklist" for pilots. The symptoms of MCAS are the same as runaway trim and the fix is the same (which is why Boeing didn't feel like extra pilot training was needed), so I'm curious to see the most recent investigation and hear the black box voice recorder. Did they not know they were dealing with runaway trim? Did they think it was something else? Did they forget the memory checklist? Was there not enough height to deal with runaway trim regardless? Were the symptoms different than runaway trim, confusing the pilots? The black boxes will be very illuminating on this respect, especially since we never recovered the Lion Air black box voice recorder.
- qyv 8y ago
  You are correct. And if two crews in 5 months had the same issue either identifying or dealing with the same problem, then perhaps there is an design or training problem error here.
  - ineedasername 8y ago
    No need to choose, it seems like design and training can share the spot light, with their third friend sensor failure.
- cybrjoe 8y ago
  A quick search indicates the CVR for Lion Air 610 was recovered. Did I miss something?
  - jquery 8y ago
    You're right, looks like it was recovered earlier this year, and they haven't released the transcript yet. https://www.reuters.com/article/us-indonesia-crash/no-public-details-on-crashed-lion-air-voice-recorder-until-final-report-indonesian-official-idUSKCN1PG08R https://www.reuters.com/article/us-indonesia-crash/no-public...
- ricardobeat 8y ago
  > [MCAS] it’s not stopped by the pilot pulling the yoke, which for normal trim from the autopilot or runaway manual trim triggers trim hold sensors This implies that 'normal' runaway trim can be stopped by pulling the control yoke. Maybe pilots simply have no idea what is going on once they realise that action has no effect with the MCAS?
flashman 8y ago
So a single point of failure (malfunctioning sensor) can engage the horizontal stabilizer without notifying pilots, in a way that the control yoke can't override. What the hell did Boeing think was going to happen?
- jimktrains2 8y ago
  Do we have a confirmed source that it can't be override by pilot input? My understanding was that using the stick could overcome it with other control surfaces and that the controls for trim can be set/reset by the pilot. I think Boeing has handled the aftermath (and much of the lead up since the release of the plane) very, very poorly. I, as a layman to aviation, am not willing to bet that Boeing knew the true likelihood of a problem and didn't tell anyone or had a whistle blower over it. However, if the issue with a non-redundant hydraulic value in the original 737 didn't teach us the lesson, this should: no matter the likelihood of failure, safety critical systems should always be redundant. (Also, Boeing didn't handle that original issue very well either.)
  - dsfyu404ed 8y ago
    >safety critical systems should always be redundant. That's east to say from an armchair but on an aircraft everything is "safety critical" to some extent and you have to choose what gets redundancy. Something where without it you can't fly the plane sure, that make sense. The argument for considering MCAS, a system which is not necessary to fly the plane safely, to be "safety critical" is much weaker. The Lion Air crash wouldn't have happened had the pilots disabled MCAS instead of fighting it into the drink.
    - ajxs 8y ago
      I'm fairly certain that the characteristic "If this component malfunctions loss of life is one potential outcome" is a solid case for the component actually being safety critical. The pilots would never have needed to disable MCAS had it not malfunctioned in this manner. I'm not sure what redundancy has to do with this, but clearly there was a failure in a safety-critical system.
amanzi 8y ago
There's a really good "The Daily" podcast (~20 minutes) about these crashes that answers a lot of the questions on this page. https://www.nytimes.com/2019/03/19/podcasts/the-daily/boeing-737-max-ethiopia-crash.html https://www.nytimes.com/2019/03/19/podcasts/the-daily/boeing...
shiven 8y ago
From my point-of-view, two opinion points: 1. I am glad that 737 MAX has been grounded. May it stay that way, globally, until this issue is provably resolved. 2. The entire Boeing chain of management that resulted in these crashes should be publicly flogged, their remuneration & benefits clawed back & subject to a mandatory minimum prison sentence. Who the hell am I kidding! Neither is very likely to happen in the present day US. Carry on then, I guess. Just make sure to sign your Last Will & Testament before taking that next flight.
- shivo 8y ago
  Very grim but very true.
- JMTQp8lwXL 8y ago
  Even if the US doesn't choose to do much (though I find it embarrassing the FAA was one of the last regulatory bodies to respond), Boeing will face a reckoning globally from other regulatory agencies. Stock is down 15% since March 1. Hard to know what an executives there are thinking, but I hope some folks in the organization genuinely feels some sort of empathy for the families of the deceased on these flights.
  - markdown 8y ago
    > though I find it embarrassing the FAA was one of the last regulatory bodies to respond The top 3 officials at FAA are unfilled, with seat-warmers there in an "acting" capacity. I wonder if that's related. https://www.faa.gov/about/key_officials/ https://www.faa.gov/about/key_officials/
    - jki275 8y ago
      Honestly, those top positions in almost any organization are often political appointments that have little to do with day to day operations. The current "actings" are generally the ones who "advise" the political appointees on how to handle things. Obviously there are some exceptions, but most bureaucracies tend to run that way. Not to comment specifically on this as FAA isn't my area, but if the secdef doesn't come to work tomorrow the undersecretary is going to take the same actions he would have. I'd imagine most of those orgs trend that way.
raihansaputra 8y ago
There's a thread on twitter with a pretty good analysis of what's happening with 737MAX. The 'Swiss Cheese' model here starts from its redesign by Boeing. https://twitter.com/trevorsumner/status/1106934362531155974 https://twitter.com/trevorsumner/status/1106934362531155974
- tluyben2 8y ago
  From all I read; this including the optional disagree indicator, I still say Boeing is should be held responsible for this: all point to economic reasons which means they decided these things and fully knew the potential consequences.
twblalock 8y ago
It also seems like this was not properly reported to safety agencies at the time, nor was it reported when that plane crashed in a subsequent flight. Say what you will about Boeing, but this could have been avoided if the airline had better safety practices. Every pilot on that plane should have been trained on the new system. And the malfunction, if reported properly, should have caused that particular plane to be grounded for a mechanical inspection. Instead it flew again and hundreds of people died because they weren’t lucky enough to have one of the pilots who was trained properly.
- inferiorhuman 8y ago
  Every pilot on that plane should have been trained on the new system. How is that supposed to work when Boeing didn't inform any of the airlines of this system?
  - kayfox 8y ago
    Its in the type certificate and the maintenance manuals.
    - inferiorhuman 8y ago
      Nope. This has been the subject of a ton of teeth gnashing by the American pilot unions (the head of American Airlines' APA union has been among the most vocal critics of Boeing in this case). In fact if you look at the ANAC's (Brazil) version of the "OPERATIONAL EVALUATION REPORT" and compare it to the rest of the world you'll find only the Brazilian version references MCAS.
  - twblalock 8y ago
    Boeing did inform them. Otherwise how would the third pilot have known what to do when he saved the plane?
    - inferiorhuman 8y ago
      Boeing did inform them. Boeing did not inform any of the airlines of MCAS until after the Lion Air crash. Otherwise how would the third pilot have known what to do when he saved the plane? The deadheading pilot recognized a problem with something controlling the stabilizer and went off script. Edit: I love the downvotes from folks who haven't actually read any of the complaints from pilots. Since reading is too hard, have a video from the Washington Post: https://www.youtube.com/watch?v=ftZ6j8onS78 https://www.youtube.com/watch?v=ftZ6j8onS78
gok 8y ago
> That extra pilot, who was seated in the cockpit jumpseat, correctly diagnosed the problem and told the crew how to disable a malfunctioning flight-control system and save the plane, according to two people familiar with Indonesia’s investigation. > The presence of a third pilot in the cockpit wasn’t contained in Indonesia’s National Transportation Safety Committee’s Nov. 28 report on the crash and hasn’t previously been reported. So the NTSC explicitly chose to exclude this and then two whistleblowers went to Bloomberg? That is fucking wild.
- Someone1234 8y ago
  It didn't need to be included in the preliminary report since it is contextual/background information. They did include in the report that on a previous flight the same sensor error occurred and the pilots resolved it by disabling auto-trim. The fact there was a third pilot there is definitely interesting, but they didn't make anyone less safe by not including it in the report. There's no bombshell here. Previous problems were well known/reported before today.
  - zaphirplane 8y ago
    This is very important, as it shows the ratio of pilots aware of the mitigation is low, and/or the stress of fighting with the the computer makes you forget you training amendments Edited to add it’s a training amendment
    - arcticbull 8y ago
      It's a single data point / anecdote. These investigations are incredibly thorough, precise, and authoritative so it makes sense they'd seek to exclude that kind of information until they knew for sure.
      - hencq 8y ago
        I'm not sure that thorough and precise implies they would seek to leave out things. In fact, one would expect the opposite.
fxfan 8y ago
The MIC is too powerful and influential over both parties. While I can sometimes like trump for not submitting to anybody - even he bows to MIC
spricket 8y ago
The fact that "AoA disagree" light and logic was an optional feature seems criminal enough to me. A sensor with no failover unless you pay for an option. Who the hell thought this was a good idea or approved it? WTF! 500 people are already dead. Boeing should be brought to the coals. It probably takes longer to go through the checklist than it does for everyone to die. I promise if the audio recordings are ever released from CVR they will be absolutely damning. Pilots trying to make it through a loss of control checklist as they dive to their doom. A lot of those checklists have 50+ steps. Imagine trying to make it through that while fighting the plane and descending at over 3x "maximum design descent rate". I'm sure the fucking alarms we're blaring and pilots cursing the system carrying them toward certain death.
- tntn 8y ago
  The checklist in question has three steps. Step two is "move stab trim to cutout," which disables automatic systems that adjust the stabilizer. The pilots in lion air had ~10 minutes to do this. It is extremely unlikely that the pilots were trying to work through the checklist. More likely they simply did not know what to do.
  - spricket 8y ago
    This doesn't make jive with disaster being averted by a third pilot. Assuming the third was totally dedicated to checklist vs preventing the plane from diving, it was only his insight that stopped the plane from going down. The MCAS system apparently increased downward trim without any speed considerations, to over 2.5 degrees in 10 seconds. I don't have the full flight control details but it sure sounds like pilots would lose control within minutes at most. In the LionAir crash the pilot reported control problems and asked to return to airport within 3 minutes, and they slammed into the ocean in 12. Not sure where you're getting this info but I'm more than sure they knew something was wrong in the last 2 minutes (while they were heading into the earth at almost the speed of sound). You really think they have ten minutes to react when by then everyone on LionAir was doomed to die?
    - fegul 8y ago
abbadadda 8y ago
What are the odds this is completely fabricated by Boeing? Not saying I think this, but if it was a movie and this was a cover-up, this would be a great plot twist. I suppose I'm maybe just a little jaded from all the fake news these days.
- jagthebeetle 8y ago
  There's a difference between jaded and cynical :) Not that I know the answer to your question any better. We'll have to wait for the HBO documentary. To play devil's advocate anyway, as someone who has not been following this actively, I find this article to cement the idea in this reader's mind that a Boeing malfunction is involved in all three incidents. Is this even conclusively established? Would Boeing want this spin at this point? The suggestion that an extra brain might randomly have averted two multi-fatal crashes and that this error mode has occurred at least three times seems like it would be a bit pyrrhic for the PR people at this juncture, no?
briandear 8y ago
> The Indonesia safety committee report said the plane had had multiple failures on previous flights and hadn’t been properly repaired. Lots of blame for Boeing, but the real criminals are Lion Air who apparently don’t know how to maintain airplanes. Compare their safety record with Southwest Airlines. Lion Air shouldn’t be allowed to fly.
cmurf 8y ago
Jump pilot would have had a natural line of sight too the trim wheel, and may have seen it move "unscheduled" at the same time as the nose down. This might have given him a unique suspicion of auto trim. I expect this will be included in the accident report. Hopefully NTSB will conduct their own first hand interview with this pilot. (I can't think of why they wouldn't.)
torqueTorrent 8y ago
The saddest thing, as many HN users should know all to well, is that there can be no excuse for automated systems like airliners to experience catastrophic failure and loss of life, if only due to the availability and application of modern SDLC principles and CI/CD etc. Smoke testing could have been performed such that all possible combinations of transducer input could be considered and evaluated thoroughly for closed-loop effect at runtime. These types of integration tests should have been performed repeatedly, seemingly endlessly in the quest for bugs and analysis of the full spectrum of runtime results and effects. In my experience in the software industry, I've always done this for applications that have infinitely more trivial effect and results than an airliner at altitude containing hundreds of souls. One potential counterpart to the seemingly infinite greed we see exponentially increasing could be the old adage that karma is a bitch.
- philpem 8y ago
  Speaking as someone who's done this (though not on a something as big as an airliner!) Yes, you can test control loops -- you can even turn it into a unit test. At least in theory. The problem is that to do the test you need either a working, physical system or a good model. So if you're making a shutdown valve for a chemical plant, you need a physical build of that control valve. Even on that scale, you're talking about something that could potentially fill an engineering lab, be quite noisy and have a considerable amount of stored pneumatic or hydraulic energy. It's possible, but not exactly practical. The alternative is to model the system, but now the question changes: how can you be certain that your model is accurate and models all the variables? Say your valve is slower when it's cold and you don't model that -- now you have a false positive result ("it works" -- but nobody realised that "temperature" was a dependent variable). So you take the middle ground - you can have the test jig for a week, so you record the inputs and outputs for a week under varying software conditions. But those recordings are only valid for that specific timing -- if you change the software and change the timing (maybe you move the trim motor slower), you get a model change and a false positive or negative. It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary.
  - 8y ago
fjfaase 8y ago
I am almost sure that some engineer of Boeing has noticed that there was a major design flaw with the function of the MCAS, but that he was overruled by a less technical (and probably younger) superior.
- kkarakk 8y ago
  more likely the flaw was noted but it would be more expensive to redesign than to eat the cost of whatever lawsuit they'd be hit with
CivBase 8y ago
> A malfunctioning sensor is believed to have tricked the Lion Air plane’s computers into thinking it needed to automatically bring the nose down to avoid a stall. That is ridiculous logic to implement in a "safety" system. An automated system should never cause a plane to dive unless it also knows that it has enough altitude to safely do so - much less in a way that makes it difficult for pilots to override.
- proaralyst 8y ago
  Given the system assumed it was in stall, which means loss of altitude anyway, surely it's safer in general to go nose-down to avoid the stall? At least then you have a chance of recovery, which you don't in a stall. (Except of course, going nose-down.)
  - CivBase 8y ago
    If you don't have enough altitude to afford going nose-down, let the pilot handle it. If your system can't come up with a safe solution, do not override the pilot's controls in favor of a dangerous solution. A computer should never assume it knows better than a pilot. A computer is only as good as the data it gets and the software it runs. Sensors fail. Data gets corrupted. In the current state of the industry, software bugs are inevitable. Airplane software is supposed to help pilots, not hinder them. In light of these events, I'm thinking twice about wanting a self-driving car in the near future.
    - cmurf 8y ago
      Elsewhere I've read MCAS does take altitude into account, as well as flaps, i.e. it's only active above a certain altitude, and only when flaps are retracted. So... yeah, we don't have the full story. And also in another thread, it's reported from the flight prior to Lion Air 610 (same plane) there were airspeed and altitude disagreements. I'm not at all clear from available reporting whether airspeed, altitude, and angle of attack were inconsistent, if that was a source of either autopilot confusion, and then pilot confusion, whether pilots did set stabilizer trim to cutoff and when and whether it was too late. I'm a pilot (former CFII) and the whole automation fail danger strikes me as terrible. John Q Public says "I want the automation to override the pilot's mistakes!" What? OK fine. What about Asiana Airlines Flight 214 where the pilot advanced throttles, an explicit intent input, and yet autothrottles were set so the automation said nope. And then John Q Public are all, well the pilot should have KNOWN! It's like it's a game where the pilot is only there as the last resort to be blamed if they too fail, even after a sequence of automation failures. Automation betraying pilots at low altitude is in my view functionally equivalent to an in-flight breakup. And automation in the cockpit mentality in the face of failures has been, for 20+years, "add another button, add another feature, add another routine" to tack on all the others. And yes this absolutely makes me think of autonomous driving as total b.s. Airplanes are in a standardized system, with far bigger budgets for automation and yet we still have to fall back to human pilots for routine procedures like parking, taxiing, VFR approaches and landings, and telling the automation literally every detail it needs to do, ATC communication. It's ripe for end to end automation and yet we still don't do that. Driving cars is wildly more complicated for automation: non-standard streets, paint, signage, laws, pedestrian behavior, bicycles, cars still driven by humans, weather - haha. Sounds nice, great idea, keep trying, but it's complete bullshit.
logfromblammo 8y ago
A machine that relies on sensors has two ways to detect when a sensor has failed: another sensor, or human observer input. I don't know how avionics hardware engineers do it, but in this neighborhood of the Internet, we don't trust inputs, and especially human user inputs. Because every unverified, unsanitized input is an attack vector for bringing down our software and the system it runs on. From what I have seen, the MCAS in the crashed planes relies on a single sensor--the AOA vane in the nose--and was almost solely responsible for catastrophic loss of altitude. This model of passenger jet has a paid upgrade option to add a second sensor, with disagreement detection. My question is why don't the yoke inputs from the pilots count as disagreement with the AOA sensor? If the yoke is consistently counteracting the action of the MCAS, why can't it disable itself automatically and illuminate a light to indicate it has failed? I'm guessing the pilots would have more time to search through manuals in-flight to clear the fault and re-enable the system than they would trying to disable it while it's stubbornly trying to crash the plane due to a single point of failure. It's not hard to adopt the defensive mindset that your users (or your professional testers) are maliciously trying to destroy your beautiful program with a combination of stupidity and cleverly designed unanticipated inputs. When hardware gets involved, one can personify Entropy as a being that is trying to destroy everything you love and kill you. How would Entropy take down a plane and kill all passengers? How about it freezes the AOA sensor in the "nose is at +90 degrees pitch" position? How do we defend against that attack vector? Pilot training? Oops! Entropy also made them forget that page out of thousands of possible pages of procedural training during the critical seconds they needed to remember it. The only way to fight Entropy is by making random events more independent, rather than causally linked in a failure cascade. I don't think this course towards blaming Boeing's lack of documentation and/or pilot training is helpful. I don't think there's any option for Boeing but to immediately recall and retrofit all aircraft to the multiple AOA-sensor option, at their expense, and refund every airline that actually paid extra for it.
ratsimihah 8y ago
It's one thing to memorize things, it's another thing to be able to use that knowledge in the right context and situation, particularly when under panic.

327 points by erict15 8y ago

samfisher83 8y ago

Suprised they don't have a lessons learned portal. Would have saved some lives.

zkms 8y ago
That's the sort of thing the NASA report system is for, though IDK the latencies involved in the system: https://en.wikipedia.org/wiki/Aviation_Safety_Reporting_System https://en.wikipedia.org/wiki/Aviation_Safety_Reporting_Syst... > The Aviation Safety Reporting System, or ASRS, is the US Federal Aviation Administration's (FAA) voluntary confidential reporting system that allows pilots and other aircraft crew members to confidentially report near misses and close calls in the interest of improving air safety.
WrtCdEvrydy 8y ago
Can you imagine if the public saw how often planes were close to disaster?
- inamberclad 8y ago
  Don't confuse apathy with unavailability. The public (and it sounds like you as well) just doesn't look. Here's NASA's safety database: https://asrs.arc.nasa.gov/ https://asrs.arc.nasa.gov/
- userbinator 8y ago
  They already can: http://avherald.com/ http://avherald.com/
JMTQp8lwXL 8y ago
Only as useful as people actually reading from such a portal.

nateabele 8y ago

> [T]hey got help from an unexpected source: an off-duty pilot who happened to be riding in the cockpit. That extra pilot, who was seated in the cockpit jumpseat [...] Am I missing something here? Isn't it normal for off-duty pilots to ride in the jump-seat?

inamberclad 8y ago
Yes. If the seat is unoccupied, airlines will usually let off duty pilots use it to get between different airports. If I'm remembering correctly, it's done for pilots from other airlines as well.
khazhou 8y ago
Sure. I don't think they're implying otherwise. The lucky coincidence is that he happened to be aware of the issue and the fix.
latch 8y ago
Well, you're missing the point of the story which has nothing to do with the frequency of off-duty pilots hitching a ride.
koolba 8y ago
Riding dead head as a pilot is normal. They do it all the time to get back home or wherever their next flight is from. Being in the right place, and happening to know exactly how to deal with what would otherwise kill the pilot and all the passengers, is incredibly fortunate.
- ruytlm 8y ago
  I suppose similar in terms of the right place a the right time would be the QF32 incident[0], where by chance there were two additional pilots in the cockpit; a check captain, and a supervising check captain who was training that check captain. [0]: https://en.wikipedia.org/wiki/Qantas_Flight_32 https://en.wikipedia.org/wiki/Qantas_Flight_32
  - 8y ago

xvf22 8y ago

An extra set of eyes saved them, it's a shame that there wasn't any way for them to include a reminder in the next crews flight plans.

sundvor 8y ago
It is pretty shocking this wasn't noted as a serious incident needing investigation before more flights were undertaken.

jnbiche 8y ago

...

jimktrains2 8y ago
Do we know if the issue is bad software or bad specs? Even formal methods won't save you from a bad spec.
- khuey 8y ago
  I don't think we've seen anything to indicate that the problem is not a bad spec.
  - jnbiche 8y ago
    We haven't seen anything either way. Even the allegation that it's the MCAS system is totally unconfirmed to the public.
- jnbiche 8y ago
  Yes, and I mentioned this in my second paragraph. And no, we the public know very little about this point about the current alleged MCAS problem, but we know that many problems with safety critical software comes from implementations, not specs.
platinumrad 8y ago
>In fact, I thought these crash recovery systems had to be written by 3 different teams in 3 different languages or architectures, and only activate when 2 of the 3 signal true. Or there's some manufacturer that does avionic software like this, I forget who. Redundant systems with majority voting are common, and this can include multiple implementations of the same spec, but I'm not aware of any manufacturer that intentionally uses three different languages or architectures.
- jimktrains2 8y ago
  I don't know about 3 languages, but https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff implies that there is software written by at least two different groups running on the shuttle. I've also heard anecdotally that some train signalling and on-board control systems run software simultaneously in linuc and freebsd to oessen the chance of simultaneous memory faults. (I didn't get a chance to ask for more details or clarification.)

reasonablemann 8y ago

Is it possible for Boeing engineers to lose their professional status as a result of this situation?

cloakandswagger 8y ago
Over pilot error and an unmaintained airplane? Doubtful.
- _jal 8y ago
  You seem very certain of the cause(s) of the crash. Care to share the source of your knowledge?
  - rootusrootus 8y ago
    Just as certain as everyone else on HN that knows it's all Boeing's fault.
    - _jal 8y ago
      Partly my point. Plenty if internet experts to go around.
- sokoloff 8y ago
  I agree that the pilots likely shoulder some of the blame (and in an NTSB-investigated crash, I'd expect their failure to follow the non-normal checklist memory items to be the primary cause), it's not enough to say that this was simple pilot error and poor maintenance. Boeing's going to wear some of the blame here, as is proper, IMO.
- ceejayoz 8y ago
  From the article: "There have been no reports of maintenance issues with the Ethiopian Airlines plane before its crash."
jimktrains2 8y ago
Wouldn't a better point of view be: shouldn't we reëvaluate the items that allow self-certification and fund the FAA properly so that they can certify medium/large change themselves instead of letting the manufacturer? Self-certification has a place, but it should always be accompanied by random checks and shouldn't be for anything large, critical, or first time through. Edit: Better in that it helps solve what is largely a political and not an engineering problem.

pimlottc 8y ago

The headline is really confusing; I thought I was perhaps reading it wrong, but it's sort of impossible to read correctly until you realize they are talking about a separate incident than the well-known crash. It would have been clearer if they had written something simpler, like "Lion Air 737 Nearly Crashed One Day Before Deadly Accident"

JMTQp8lwXL 8y ago
Agreed. I ended up reading a second article on the topic to ensure I was understanding it correctly (the prior day's flight involving the same identical aircraft, not a different one of the same model).
system2 8y ago
I read it 5 times and I am glad I wasn't the one. I thought it was 737 days before.
foobarbecue 8y ago
I still can't tell if it says whether or was actually the same individual aircraft.
- RyJones 8y ago
  Same aircraft
dang 8y ago
We added "on the" to try to make that clearer above.

jquery 8y ago

Runaway trim is supposed to be part of the "memory checklist" for pilots. The symptoms of MCAS are the same as runaway trim and the fix is the same (which is why Boeing didn't feel like extra pilot training was needed), so I'm curious to see the most recent investigation and hear the black box voice recorder. Did they not know they were dealing with runaway trim? Did they think it was something else? Did they forget the memory checklist? Was there not enough height to deal with runaway trim regardless? Were the symptoms different than runaway trim, confusing the pilots? The black boxes will be very illuminating on this respect, especially since we never recovered the Lion Air black box voice recorder.

qyv 8y ago
You are correct. And if two crews in 5 months had the same issue either identifying or dealing with the same problem, then perhaps there is an design or training problem error here.
- ineedasername 8y ago
  No need to choose, it seems like design and training can share the spot light, with their third friend sensor failure.
cybrjoe 8y ago
A quick search indicates the CVR for Lion Air 610 was recovered. Did I miss something?
- jquery 8y ago
  You're right, looks like it was recovered earlier this year, and they haven't released the transcript yet. https://www.reuters.com/article/us-indonesia-crash/no-public-details-on-crashed-lion-air-voice-recorder-until-final-report-indonesian-official-idUSKCN1PG08R https://www.reuters.com/article/us-indonesia-crash/no-public...
ricardobeat 8y ago
> [MCAS] it’s not stopped by the pilot pulling the yoke, which for normal trim from the autopilot or runaway manual trim triggers trim hold sensors This implies that 'normal' runaway trim can be stopped by pulling the control yoke. Maybe pilots simply have no idea what is going on once they realise that action has no effect with the MCAS?

flashman 8y ago

So a single point of failure (malfunctioning sensor) can engage the horizontal stabilizer without notifying pilots, in a way that the control yoke can't override. What the hell did Boeing think was going to happen?

jimktrains2 8y ago
Do we have a confirmed source that it can't be override by pilot input? My understanding was that using the stick could overcome it with other control surfaces and that the controls for trim can be set/reset by the pilot. I think Boeing has handled the aftermath (and much of the lead up since the release of the plane) very, very poorly. I, as a layman to aviation, am not willing to bet that Boeing knew the true likelihood of a problem and didn't tell anyone or had a whistle blower over it. However, if the issue with a non-redundant hydraulic value in the original 737 didn't teach us the lesson, this should: no matter the likelihood of failure, safety critical systems should always be redundant. (Also, Boeing didn't handle that original issue very well either.)
- dsfyu404ed 8y ago
  >safety critical systems should always be redundant. That's east to say from an armchair but on an aircraft everything is "safety critical" to some extent and you have to choose what gets redundancy. Something where without it you can't fly the plane sure, that make sense. The argument for considering MCAS, a system which is not necessary to fly the plane safely, to be "safety critical" is much weaker. The Lion Air crash wouldn't have happened had the pilots disabled MCAS instead of fighting it into the drink.
  - ajxs 8y ago
    I'm fairly certain that the characteristic "If this component malfunctions loss of life is one potential outcome" is a solid case for the component actually being safety critical. The pilots would never have needed to disable MCAS had it not malfunctioned in this manner. I'm not sure what redundancy has to do with this, but clearly there was a failure in a safety-critical system.

amanzi 8y ago

There's a really good "The Daily" podcast (~20 minutes) about these crashes that answers a lot of the questions on this page. https://www.nytimes.com/2019/03/19/podcasts/the-daily/boeing-737-max-ethiopia-crash.html https://www.nytimes.com/2019/03/19/podcasts/the-daily/boeing...

shiven 8y ago

From my point-of-view, two opinion points: 1. I am glad that 737 MAX has been grounded. May it stay that way, globally, until this issue is provably resolved. 2. The entire Boeing chain of management that resulted in these crashes should be publicly flogged, their remuneration & benefits clawed back & subject to a mandatory minimum prison sentence. Who the hell am I kidding! Neither is very likely to happen in the present day US. Carry on then, I guess. Just make sure to sign your Last Will & Testament before taking that next flight.

shivo 8y ago
Very grim but very true.
JMTQp8lwXL 8y ago
Even if the US doesn't choose to do much (though I find it embarrassing the FAA was one of the last regulatory bodies to respond), Boeing will face a reckoning globally from other regulatory agencies. Stock is down 15% since March 1. Hard to know what an executives there are thinking, but I hope some folks in the organization genuinely feels some sort of empathy for the families of the deceased on these flights.
- markdown 8y ago
  > though I find it embarrassing the FAA was one of the last regulatory bodies to respond The top 3 officials at FAA are unfilled, with seat-warmers there in an "acting" capacity. I wonder if that's related. https://www.faa.gov/about/key_officials/ https://www.faa.gov/about/key_officials/
  - jki275 8y ago
    Honestly, those top positions in almost any organization are often political appointments that have little to do with day to day operations. The current "actings" are generally the ones who "advise" the political appointees on how to handle things. Obviously there are some exceptions, but most bureaucracies tend to run that way. Not to comment specifically on this as FAA isn't my area, but if the secdef doesn't come to work tomorrow the undersecretary is going to take the same actions he would have. I'd imagine most of those orgs trend that way.

raihansaputra 8y ago

There's a thread on twitter with a pretty good analysis of what's happening with 737MAX. The 'Swiss Cheese' model here starts from its redesign by Boeing. https://twitter.com/trevorsumner/status/1106934362531155974 https://twitter.com/trevorsumner/status/1106934362531155974

tluyben2 8y ago
From all I read; this including the optional disagree indicator, I still say Boeing is should be held responsible for this: all point to economic reasons which means they decided these things and fully knew the potential consequences.

twblalock 8y ago

It also seems like this was not properly reported to safety agencies at the time, nor was it reported when that plane crashed in a subsequent flight. Say what you will about Boeing, but this could have been avoided if the airline had better safety practices. Every pilot on that plane should have been trained on the new system. And the malfunction, if reported properly, should have caused that particular plane to be grounded for a mechanical inspection. Instead it flew again and hundreds of people died because they weren’t lucky enough to have one of the pilots who was trained properly.

inferiorhuman 8y ago
Every pilot on that plane should have been trained on the new system. How is that supposed to work when Boeing didn't inform any of the airlines of this system?
- kayfox 8y ago
  Its in the type certificate and the maintenance manuals.
  - inferiorhuman 8y ago
    Nope. This has been the subject of a ton of teeth gnashing by the American pilot unions (the head of American Airlines' APA union has been among the most vocal critics of Boeing in this case). In fact if you look at the ANAC's (Brazil) version of the "OPERATIONAL EVALUATION REPORT" and compare it to the rest of the world you'll find only the Brazilian version references MCAS.
- twblalock 8y ago
  Boeing did inform them. Otherwise how would the third pilot have known what to do when he saved the plane?
  - inferiorhuman 8y ago
    Boeing did inform them. Boeing did not inform any of the airlines of MCAS until after the Lion Air crash. Otherwise how would the third pilot have known what to do when he saved the plane? The deadheading pilot recognized a problem with something controlling the stabilizer and went off script. Edit: I love the downvotes from folks who haven't actually read any of the complaints from pilots. Since reading is too hard, have a video from the Washington Post: https://www.youtube.com/watch?v=ftZ6j8onS78 https://www.youtube.com/watch?v=ftZ6j8onS78

gok 8y ago

> That extra pilot, who was seated in the cockpit jumpseat, correctly diagnosed the problem and told the crew how to disable a malfunctioning flight-control system and save the plane, according to two people familiar with Indonesia’s investigation. > The presence of a third pilot in the cockpit wasn’t contained in Indonesia’s National Transportation Safety Committee’s Nov. 28 report on the crash and hasn’t previously been reported. So the NTSC explicitly chose to exclude this and then two whistleblowers went to Bloomberg? That is fucking wild.

Someone1234 8y ago
It didn't need to be included in the preliminary report since it is contextual/background information. They did include in the report that on a previous flight the same sensor error occurred and the pilots resolved it by disabling auto-trim. The fact there was a third pilot there is definitely interesting, but they didn't make anyone less safe by not including it in the report. There's no bombshell here. Previous problems were well known/reported before today.
- zaphirplane 8y ago
  This is very important, as it shows the ratio of pilots aware of the mitigation is low, and/or the stress of fighting with the the computer makes you forget you training amendments Edited to add it’s a training amendment
  - arcticbull 8y ago
    It's a single data point / anecdote. These investigations are incredibly thorough, precise, and authoritative so it makes sense they'd seek to exclude that kind of information until they knew for sure.
    - hencq 8y ago
      I'm not sure that thorough and precise implies they would seek to leave out things. In fact, one would expect the opposite.

fxfan 8y ago

The MIC is too powerful and influential over both parties. While I can sometimes like trump for not submitting to anybody - even he bows to MIC

spricket 8y ago

The fact that "AoA disagree" light and logic was an optional feature seems criminal enough to me. A sensor with no failover unless you pay for an option. Who the hell thought this was a good idea or approved it? WTF! 500 people are already dead. Boeing should be brought to the coals. It probably takes longer to go through the checklist than it does for everyone to die. I promise if the audio recordings are ever released from CVR they will be absolutely damning. Pilots trying to make it through a loss of control checklist as they dive to their doom. A lot of those checklists have 50+ steps. Imagine trying to make it through that while fighting the plane and descending at over 3x "maximum design descent rate". I'm sure the fucking alarms we're blaring and pilots cursing the system carrying them toward certain death.

tntn 8y ago
The checklist in question has three steps. Step two is "move stab trim to cutout," which disables automatic systems that adjust the stabilizer. The pilots in lion air had ~10 minutes to do this. It is extremely unlikely that the pilots were trying to work through the checklist. More likely they simply did not know what to do.
- spricket 8y ago
  This doesn't make jive with disaster being averted by a third pilot. Assuming the third was totally dedicated to checklist vs preventing the plane from diving, it was only his insight that stopped the plane from going down. The MCAS system apparently increased downward trim without any speed considerations, to over 2.5 degrees in 10 seconds. I don't have the full flight control details but it sure sounds like pilots would lose control within minutes at most. In the LionAir crash the pilot reported control problems and asked to return to airport within 3 minutes, and they slammed into the ocean in 12. Not sure where you're getting this info but I'm more than sure they knew something was wrong in the last 2 minutes (while they were heading into the earth at almost the speed of sound). You really think they have ten minutes to react when by then everyone on LionAir was doomed to die?
  - fegul 8y ago

abbadadda 8y ago

What are the odds this is completely fabricated by Boeing? Not saying I think this, but if it was a movie and this was a cover-up, this would be a great plot twist. I suppose I'm maybe just a little jaded from all the fake news these days.

jagthebeetle 8y ago
There's a difference between jaded and cynical :) Not that I know the answer to your question any better. We'll have to wait for the HBO documentary. To play devil's advocate anyway, as someone who has not been following this actively, I find this article to cement the idea in this reader's mind that a Boeing malfunction is involved in all three incidents. Is this even conclusively established? Would Boeing want this spin at this point? The suggestion that an extra brain might randomly have averted two multi-fatal crashes and that this error mode has occurred at least three times seems like it would be a bit pyrrhic for the PR people at this juncture, no?

briandear 8y ago

> The Indonesia safety committee report said the plane had had multiple failures on previous flights and hadn’t been properly repaired. Lots of blame for Boeing, but the real criminals are Lion Air who apparently don’t know how to maintain airplanes. Compare their safety record with Southwest Airlines. Lion Air shouldn’t be allowed to fly.

cmurf 8y ago

Jump pilot would have had a natural line of sight too the trim wheel, and may have seen it move "unscheduled" at the same time as the nose down. This might have given him a unique suspicion of auto trim. I expect this will be included in the accident report. Hopefully NTSB will conduct their own first hand interview with this pilot. (I can't think of why they wouldn't.)

torqueTorrent 8y ago

The saddest thing, as many HN users should know all to well, is that there can be no excuse for automated systems like airliners to experience catastrophic failure and loss of life, if only due to the availability and application of modern SDLC principles and CI/CD etc. Smoke testing could have been performed such that all possible combinations of transducer input could be considered and evaluated thoroughly for closed-loop effect at runtime. These types of integration tests should have been performed repeatedly, seemingly endlessly in the quest for bugs and analysis of the full spectrum of runtime results and effects. In my experience in the software industry, I've always done this for applications that have infinitely more trivial effect and results than an airliner at altitude containing hundreds of souls. One potential counterpart to the seemingly infinite greed we see exponentially increasing could be the old adage that karma is a bitch.

philpem 8y ago
Speaking as someone who's done this (though not on a something as big as an airliner!) Yes, you can test control loops -- you can even turn it into a unit test. At least in theory. The problem is that to do the test you need either a working, physical system or a good model. So if you're making a shutdown valve for a chemical plant, you need a physical build of that control valve. Even on that scale, you're talking about something that could potentially fill an engineering lab, be quite noisy and have a considerable amount of stored pneumatic or hydraulic energy. It's possible, but not exactly practical. The alternative is to model the system, but now the question changes: how can you be certain that your model is accurate and models all the variables? Say your valve is slower when it's cold and you don't model that -- now you have a false positive result ("it works" -- but nobody realised that "temperature" was a dependent variable). So you take the middle ground - you can have the test jig for a week, so you record the inputs and outputs for a week under varying software conditions. But those recordings are only valid for that specific timing -- if you change the software and change the timing (maybe you move the trim motor slower), you get a model change and a false positive or negative. It's certainly possible, but it's only possible with a good sized team, and supportive management who realise that the test is absolutely necessary.
- 8y ago

fjfaase 8y ago

I am almost sure that some engineer of Boeing has noticed that there was a major design flaw with the function of the MCAS, but that he was overruled by a less technical (and probably younger) superior.

kkarakk 8y ago
more likely the flaw was noted but it would be more expensive to redesign than to eat the cost of whatever lawsuit they'd be hit with

CivBase 8y ago

> A malfunctioning sensor is believed to have tricked the Lion Air plane’s computers into thinking it needed to automatically bring the nose down to avoid a stall. That is ridiculous logic to implement in a "safety" system. An automated system should never cause a plane to dive unless it also knows that it has enough altitude to safely do so - much less in a way that makes it difficult for pilots to override.

proaralyst 8y ago
Given the system assumed it was in stall, which means loss of altitude anyway, surely it's safer in general to go nose-down to avoid the stall? At least then you have a chance of recovery, which you don't in a stall. (Except of course, going nose-down.)
- CivBase 8y ago
  If you don't have enough altitude to afford going nose-down, let the pilot handle it. If your system can't come up with a safe solution, do not override the pilot's controls in favor of a dangerous solution. A computer should never assume it knows better than a pilot. A computer is only as good as the data it gets and the software it runs. Sensors fail. Data gets corrupted. In the current state of the industry, software bugs are inevitable. Airplane software is supposed to help pilots, not hinder them. In light of these events, I'm thinking twice about wanting a self-driving car in the near future.
  - cmurf 8y ago
    Elsewhere I've read MCAS does take altitude into account, as well as flaps, i.e. it's only active above a certain altitude, and only when flaps are retracted. So... yeah, we don't have the full story. And also in another thread, it's reported from the flight prior to Lion Air 610 (same plane) there were airspeed and altitude disagreements. I'm not at all clear from available reporting whether airspeed, altitude, and angle of attack were inconsistent, if that was a source of either autopilot confusion, and then pilot confusion, whether pilots did set stabilizer trim to cutoff and when and whether it was too late. I'm a pilot (former CFII) and the whole automation fail danger strikes me as terrible. John Q Public says "I want the automation to override the pilot's mistakes!" What? OK fine. What about Asiana Airlines Flight 214 where the pilot advanced throttles, an explicit intent input, and yet autothrottles were set so the automation said nope. And then John Q Public are all, well the pilot should have KNOWN! It's like it's a game where the pilot is only there as the last resort to be blamed if they too fail, even after a sequence of automation failures. Automation betraying pilots at low altitude is in my view functionally equivalent to an in-flight breakup. And automation in the cockpit mentality in the face of failures has been, for 20+years, "add another button, add another feature, add another routine" to tack on all the others. And yes this absolutely makes me think of autonomous driving as total b.s. Airplanes are in a standardized system, with far bigger budgets for automation and yet we still have to fall back to human pilots for routine procedures like parking, taxiing, VFR approaches and landings, and telling the automation literally every detail it needs to do, ATC communication. It's ripe for end to end automation and yet we still don't do that. Driving cars is wildly more complicated for automation: non-standard streets, paint, signage, laws, pedestrian behavior, bicycles, cars still driven by humans, weather - haha. Sounds nice, great idea, keep trying, but it's complete bullshit.

logfromblammo 8y ago

A machine that relies on sensors has two ways to detect when a sensor has failed: another sensor, or human observer input. I don't know how avionics hardware engineers do it, but in this neighborhood of the Internet, we don't trust inputs, and especially human user inputs. Because every unverified, unsanitized input is an attack vector for bringing down our software and the system it runs on. From what I have seen, the MCAS in the crashed planes relies on a single sensor--the AOA vane in the nose--and was almost solely responsible for catastrophic loss of altitude. This model of passenger jet has a paid upgrade option to add a second sensor, with disagreement detection. My question is why don't the yoke inputs from the pilots count as disagreement with the AOA sensor? If the yoke is consistently counteracting the action of the MCAS, why can't it disable itself automatically and illuminate a light to indicate it has failed? I'm guessing the pilots would have more time to search through manuals in-flight to clear the fault and re-enable the system than they would trying to disable it while it's stubbornly trying to crash the plane due to a single point of failure. It's not hard to adopt the defensive mindset that your users (or your professional testers) are maliciously trying to destroy your beautiful program with a combination of stupidity and cleverly designed unanticipated inputs. When hardware gets involved, one can personify Entropy as a being that is trying to destroy everything you love and kill you. How would Entropy take down a plane and kill all passengers? How about it freezes the AOA sensor in the "nose is at +90 degrees pitch" position? How do we defend against that attack vector? Pilot training? Oops! Entropy also made them forget that page out of thousands of possible pages of procedural training during the critical seconds they needed to remember it. The only way to fight Entropy is by making random events more independent, rather than causally linked in a failure cascade. I don't think this course towards blaming Boeing's lack of documentation and/or pilot training is helpful. I don't think there's any option for Boeing but to immediately recall and retrofit all aircraft to the multiple AOA-sensor option, at their expense, and refund every airline that actually paid extra for it.

ratsimihah 8y ago

It's one thing to memorize things, it's another thing to be able to use that knowledge in the right context and situation, particularly when under panic.