8 ms·

I'm just amazed by how poorly this was implemented. Software engineers expect hardware to fail. This wasn't just the work of a newb (or I hope it wasn't). This

I'm just amazed by how poorly this was implemented. Software engineers expect hardware to fail. This wasn't just the work of a newb (or I hope it wasn't). This was planned; both the requirements and the failure modes. Then implemented by another team, and the quality verified by another team, even a 3rd party. There wasn't even a cutoff if it started operating outside it's expected range, surely one of those people would have noticed the shortfalls. This software should have the same level of engineering as other systems/ parts of the plane. Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent. This software wasn't even a crucial feature. It was only needed when high power is applied to the engines (like during takeoff) to stop the nose feeling too light by existing 737 pilots. This wasn't a jet-fighter, which would become unstable without the aid of computers. It's a shame 338 died because of what is effectively a 'schoolboy error'.

js2 8y ago
> Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Pitch != AoA. https://aviation.stackexchange.com/questions/2287/can-the-pitch-be-very-different-from-the-angle-of-attack https://aviation.stackexchange.com/questions/2287/can-the-pi... http://www.aerospaceweb.org/question/aerodynamics/q0165.shtml http://www.aerospaceweb.org/question/aerodynamics/q0165.shtm...
yutyut 8y ago
>Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! You would need a sensor to measure relative wind. Angle of attack is the angle between the wing and the relative wind.
- flyinglizard 8y ago
  A six axis sensor could probably measure AoA in a pinch. It’s just like how a stability control system works in a car (which is measuring slip angle, which is the lateral “angle of attack” between tires and road). If you detect upwards rotation but no matching acceleration forces, you’re detaching from the air stream. Obviously this system will drift over time, but it could more than adequate to detect a failed AoA sensor and revert to the second one.
  - mjevans 8y ago
    It could at least provide a reference to infer if readings were within a sensible range. Outside of that range it might be best to scream bloody murder and require manual human piloting with automated assistance at producing the desired results. (E.G. If the pilot's pulling back on the stick, they PROBABLY want the airframe to go up as a result rather than down.)
  - bbojan 8y ago
    Since angle of attack is by definition the angle between a reference line of a body and the fluid it is moving through, I fail to see how you could measure it without measuring the relative motion of the fluid. Your suggestion would work under the assumption that there is never any vertical motion of air, but that's not the case in real life. E.g. in a downdraft your sensor would fail to detect a stall.
    - flyinglizard 8y ago
      That’s true, but it would still be good enough to cross check the aerodynamic AoA sensors upon a disagree between the two and select the one that’s more reliable.
      - jedmeyers 8y ago
        > it would still be good enough to cross check What are you basing this assumption on?
peteradio 8y ago
This is a weather vane which needs to be exposed to the elements, its not something that can be inside the aircraft. It needs to be exposed to super harsh conditions to get its reading.
- lutorm 8y ago
  Except you can estimate the Aoa pretty well just using airspeed and load factor.
  - tobib 8y ago
    Can you elaborate please?
  - esoterae 8y ago
    Estimates don't fly in aviation.
    - lutorm 8y ago
      That's a ridiculous statement. All sensor data are estimates. If the IMU and airspeed sensors are good enough to feed the artificial horizon, autopilot, etc, they're surely good enough to produce an AoA estimate that's good enough to serve as a cross-check on the physical sensor.
      - peteradio 8y ago
        > That's a ridiculous statement. All sensor data are estimates. Some more reliable than others. You have no fucking clue what direction the wind is coming down on the wings based on horizon. You have a skewed perception of what wind is. On the ground it is always horizontal because it can't go into the ground. Once you get up into the sky the dynamics are insane. Get out of your armchair.
        lutorm 8y ago
        Are you a pilot? This is the basic way that a pilot can know whether he's close to stall without anything more than an airspeed indicator. It's basic flight physics. The lift developed by the wing is equal to CL.q.A. A is the wing area, the dynamic pressure q is basically indicated by the airspeed indicator, and CL is a known function of AoA. Lift is by definition equal to the load factor * mass. For a good, in-depth explanation, see http://www.av8n.com/how/htm/aoa.html#sec-ias-aoa http://www.av8n.com/how/htm/aoa.html#sec-ias-aoa
inflatableDodo 8y ago
Here's a version of how it occured; "Instead, engineers were able to add just a few inches to the front landing gear and shift the engines farther forward on the wing. The engines fit, but the Max sat at a slightly uneven angle when parked. While that design solved one problem, it created another. The larger size and new location of the engines gave the Max the tendency to tilt up during certain flight maneuvers, potentially to a dangerous angle. To compensate, Boeing engineers created the automated anti-stall system, called MCAS, that pushed the jet’s nose down if it was lifting too high. The software was intended to operate in the background so that the Max flew just like its predecessor. Boeing didn’t mention the system in its training materials for the Max. Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right. Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes." https://www.nytimes.com/2019/04/08/business/boeing-737-max-.html https://www.nytimes.com/2019/04/08/business/boeing-737-max-....
- breakingcups 8y ago
  "Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right." Shouldn't this have been more of a red flag not to rely on a single sensor when human lives are literally at stake?
  - phire 8y ago
    I suspect their reasoning was: It only trims down by 0.6°, what does it matter if a faulty sensor accdentally activates it. The plane is still flyable slightly out of trim and the pilots can easily correct it. The 0.6° number was what they said immediately after the Lion Air crash. But their reasoning was wrong. Turns out, MCAS was actually adjusting trim down by more like 1.5° and they never considered that a faulty sensor would trigger multiple MCAS activations and move the stablizer all the way to the end of it's range.
Obi_Juan_Kenobi 8y ago
GPS info might be something you can use for sanity-checking input, but AoA can only be reliably read from an actual instrument. Air moves, you can't rely on the plane's motion. Also, MCAS isn't active with flaps extended, so "like during takeoff" isn't correct. MCAS won't be active during takeoff and landing because it's not designed to be active at those times.
- velox_io 8y ago
  That's what I was getting at. If the nose is at an extreme angle (or reported to be) and the plane isn't climbing/ or descending - there's something wrong! Just after takeoff while the plane is still climbing. I'm not sure that helps matters much, as the plane is off the ground and lore's the pilots into a false sense of security as plane's working ok.. Here's the bulletin Boeing issued after the first crash. It's hidden away in the bottom left and not highlighted, so it's easy to miss: "runaway stabilizer". Unfortunately, the brief is more focussed on assuring how safe the planes are, and what counter-measures to take in the event. Rather than what a runaway event looks like. https://theaircurrent.com/aviation-safety/boeing-nearing-737-max-fleet-bulletin-on-aoa-warning-after-lion-air-crash/ https://theaircurrent.com/aviation-safety/boeing-nearing-737...
asynchronous13 8y ago
> This software wasn't even a critical feature. There are two crashes directly related to this feature. So, it doesn't make sense to claim it wasn't critical. In Boeing's own hazard analysis, they rated the feature "Hazardous", which is the second most critical rating. Obviously, it should have been rated "Catastrophic". But even with their own policy, anything rated hazardous is not allowed to have single-sensor failure. By their own policy, this should have had at least two sensors. There were multiple failures that let this slip through to production.
- velox_io 8y ago
  Sorry, I meant to write crucial (similar word, quite a different meaning..) I believe the reason for MCAS was because the had to lower and move the engines forward. So at lower power, the forward engines make the plane nose-heavy. When the engines are near full power, the lower engines push the nose up. Compounding the difference in handling compared to the older models. Here's the safety bulletin Boeing issued after the first crash. Even when you know what you're looking for it's hard to spot: "runaway stabilizer". It does seem they downplayed how serious the failure is, but they couldn't know how common it could be. That raises another question: Other pilots must have experienced the runaway event, but managed to gain control. What happened to those reports? There will be lessons learnt from these crashes, in a number of fields. [https://theaircurrent.com/aviation-safety/boeing-nearing-737-max-fleet-bulletin-on-aoa-warning-after-lion-air-crash/ https://theaircurrent.com/aviation-safety/boeing-nearing-737...]
  - chipsa 8y ago
    > I believe the reason for MCAS was because the had to lower and move the engines forward. So at lower power, the forward engines make the plane nose-heavy. When the engines are near full power, the lower engines push the nose up. Compounding the difference in handling compared to the older models. Wrong. If the power of the engines mattered, it'd have been an input to MCAS. Moving the engines forward changed how the nacelles created lift at high angles of attack, causing a pitch up effect from the nacelles, reducing (but not eliminating) the stick forces required to maintain the high angle of attack.
paulmd 8y ago
Ah yes, the classic "software engineer reveals one easy trick to solve a complex problem domain" post. Maybe they just needed a Raspberry Pi and a sixaxis sensor board, has anybody thought of that!?
- joezydeco 8y ago
  Phil Greenspun thinks it’s solvable in 10 bytes of object code! Why would you waste $35?
danaliv 8y ago
Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Unless you’ve launched your phone into the air in a fit of rage, it doesn’t have an angle of attack. :) Angle of attack is the angle between the wing and the oncoming air. You might be wondering how that’s different from how high the nose is, and you’d be justified in wondering that. Every student pilot in the world gets these two very different things mixed up! But they are very different. Next time you’re near an airport, watch the planes as they come in for landing. Try to get a vantage point from the side if you can. You’ll notice two things: (1) the airplane is descending, and (2) the nose is flat or pointed just slightly up. It should be clear that in this situation, the angle between the oncoming air and the wing — the angle of attack — and the angle of the nose are different. Because the airplane is descending, the angle of attack is greater than the angle of the nose.
monksy 8y ago
> Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent. I think this is a good point on why Agile is so dangerous to the users. Agile allows for engineering to be micromanaged and due diligence from engineers to be overlooked. (It's not something the business people care about) Don't deliver without the safety feature? Well, you're underperforming... they're going to be let go pretty quickly.

js2 8y ago

> Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Pitch != AoA. https://aviation.stackexchange.com/questions/2287/can-the-pitch-be-very-different-from-the-angle-of-attack https://aviation.stackexchange.com/questions/2287/can-the-pi... http://www.aerospaceweb.org/question/aerodynamics/q0165.shtml http://www.aerospaceweb.org/question/aerodynamics/q0165.shtm...

yutyut 8y ago

>Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! You would need a sensor to measure relative wind. Angle of attack is the angle between the wing and the relative wind.

flyinglizard 8y ago
A six axis sensor could probably measure AoA in a pinch. It’s just like how a stability control system works in a car (which is measuring slip angle, which is the lateral “angle of attack” between tires and road). If you detect upwards rotation but no matching acceleration forces, you’re detaching from the air stream. Obviously this system will drift over time, but it could more than adequate to detect a failed AoA sensor and revert to the second one.
- mjevans 8y ago
  It could at least provide a reference to infer if readings were within a sensible range. Outside of that range it might be best to scream bloody murder and require manual human piloting with automated assistance at producing the desired results. (E.G. If the pilot's pulling back on the stick, they PROBABLY want the airframe to go up as a result rather than down.)
- bbojan 8y ago
  Since angle of attack is by definition the angle between a reference line of a body and the fluid it is moving through, I fail to see how you could measure it without measuring the relative motion of the fluid. Your suggestion would work under the assumption that there is never any vertical motion of air, but that's not the case in real life. E.g. in a downdraft your sensor would fail to detect a stall.
  - flyinglizard 8y ago
    That’s true, but it would still be good enough to cross check the aerodynamic AoA sensors upon a disagree between the two and select the one that’s more reliable.
    - jedmeyers 8y ago
      > it would still be good enough to cross check What are you basing this assumption on?

peteradio 8y ago

This is a weather vane which needs to be exposed to the elements, its not something that can be inside the aircraft. It needs to be exposed to super harsh conditions to get its reading.

lutorm 8y ago
Except you can estimate the Aoa pretty well just using airspeed and load factor.
- tobib 8y ago
  Can you elaborate please?
- esoterae 8y ago
  Estimates don't fly in aviation.
  - lutorm 8y ago
    That's a ridiculous statement. All sensor data are estimates. If the IMU and airspeed sensors are good enough to feed the artificial horizon, autopilot, etc, they're surely good enough to produce an AoA estimate that's good enough to serve as a cross-check on the physical sensor.
    - peteradio 8y ago
      > That's a ridiculous statement. All sensor data are estimates. Some more reliable than others. You have no fucking clue what direction the wind is coming down on the wings based on horizon. You have a skewed perception of what wind is. On the ground it is always horizontal because it can't go into the ground. Once you get up into the sky the dynamics are insane. Get out of your armchair.
      - lutorm 8y ago
        Are you a pilot? This is the basic way that a pilot can know whether he's close to stall without anything more than an airspeed indicator. It's basic flight physics. The lift developed by the wing is equal to CL.q.A. A is the wing area, the dynamic pressure q is basically indicated by the airspeed indicator, and CL is a known function of AoA. Lift is by definition equal to the load factor * mass. For a good, in-depth explanation, see http://www.av8n.com/how/htm/aoa.html#sec-ias-aoa http://www.av8n.com/how/htm/aoa.html#sec-ias-aoa

inflatableDodo 8y ago

Here's a version of how it occured; "Instead, engineers were able to add just a few inches to the front landing gear and shift the engines farther forward on the wing. The engines fit, but the Max sat at a slightly uneven angle when parked. While that design solved one problem, it created another. The larger size and new location of the engines gave the Max the tendency to tilt up during certain flight maneuvers, potentially to a dangerous angle. To compensate, Boeing engineers created the automated anti-stall system, called MCAS, that pushed the jet’s nose down if it was lifting too high. The software was intended to operate in the background so that the Max flew just like its predecessor. Boeing didn’t mention the system in its training materials for the Max. Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right. Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes." https://www.nytimes.com/2019/04/08/business/boeing-737-max-.html https://www.nytimes.com/2019/04/08/business/boeing-737-max-....

breakingcups 8y ago
"Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right." Shouldn't this have been more of a red flag not to rely on a single sensor when human lives are literally at stake?
- phire 8y ago
  I suspect their reasoning was: It only trims down by 0.6°, what does it matter if a faulty sensor accdentally activates it. The plane is still flyable slightly out of trim and the pilots can easily correct it. The 0.6° number was what they said immediately after the Lion Air crash. But their reasoning was wrong. Turns out, MCAS was actually adjusting trim down by more like 1.5° and they never considered that a faulty sensor would trigger multiple MCAS activations and move the stablizer all the way to the end of it's range.

Obi_Juan_Kenobi 8y ago

GPS info might be something you can use for sanity-checking input, but AoA can only be reliably read from an actual instrument. Air moves, you can't rely on the plane's motion. Also, MCAS isn't active with flaps extended, so "like during takeoff" isn't correct. MCAS won't be active during takeoff and landing because it's not designed to be active at those times.

velox_io 8y ago
That's what I was getting at. If the nose is at an extreme angle (or reported to be) and the plane isn't climbing/ or descending - there's something wrong! Just after takeoff while the plane is still climbing. I'm not sure that helps matters much, as the plane is off the ground and lore's the pilots into a false sense of security as plane's working ok.. Here's the bulletin Boeing issued after the first crash. It's hidden away in the bottom left and not highlighted, so it's easy to miss: "runaway stabilizer". Unfortunately, the brief is more focussed on assuring how safe the planes are, and what counter-measures to take in the event. Rather than what a runaway event looks like. https://theaircurrent.com/aviation-safety/boeing-nearing-737-max-fleet-bulletin-on-aoa-warning-after-lion-air-crash/ https://theaircurrent.com/aviation-safety/boeing-nearing-737...

asynchronous13 8y ago

> This software wasn't even a critical feature. There are two crashes directly related to this feature. So, it doesn't make sense to claim it wasn't critical. In Boeing's own hazard analysis, they rated the feature "Hazardous", which is the second most critical rating. Obviously, it should have been rated "Catastrophic". But even with their own policy, anything rated hazardous is not allowed to have single-sensor failure. By their own policy, this should have had at least two sensors. There were multiple failures that let this slip through to production.

velox_io 8y ago
Sorry, I meant to write crucial (similar word, quite a different meaning..) I believe the reason for MCAS was because the had to lower and move the engines forward. So at lower power, the forward engines make the plane nose-heavy. When the engines are near full power, the lower engines push the nose up. Compounding the difference in handling compared to the older models. Here's the safety bulletin Boeing issued after the first crash. Even when you know what you're looking for it's hard to spot: "runaway stabilizer". It does seem they downplayed how serious the failure is, but they couldn't know how common it could be. That raises another question: Other pilots must have experienced the runaway event, but managed to gain control. What happened to those reports? There will be lessons learnt from these crashes, in a number of fields. [https://theaircurrent.com/aviation-safety/boeing-nearing-737-max-fleet-bulletin-on-aoa-warning-after-lion-air-crash/ https://theaircurrent.com/aviation-safety/boeing-nearing-737...]
- chipsa 8y ago
  > I believe the reason for MCAS was because the had to lower and move the engines forward. So at lower power, the forward engines make the plane nose-heavy. When the engines are near full power, the lower engines push the nose up. Compounding the difference in handling compared to the older models. Wrong. If the power of the engines mattered, it'd have been an input to MCAS. Moving the engines forward changed how the nacelles created lift at high angles of attack, causing a pitch up effect from the nacelles, reducing (but not eliminating) the stick forces required to maintain the high angle of attack.

paulmd 8y ago

Ah yes, the classic "software engineer reveals one easy trick to solve a complex problem domain" post. Maybe they just needed a Raspberry Pi and a sixaxis sensor board, has anybody thought of that!?

joezydeco 8y ago
Phil Greenspun thinks it’s solvable in 10 bytes of object code! Why would you waste $35?

danaliv 8y ago

Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Unless you’ve launched your phone into the air in a fit of rage, it doesn’t have an angle of attack. :) Angle of attack is the angle between the wing and the oncoming air. You might be wondering how that’s different from how high the nose is, and you’d be justified in wondering that. Every student pilot in the world gets these two very different things mixed up! But they are very different. Next time you’re near an airport, watch the planes as they come in for landing. Try to get a vantage point from the side if you can. You’ll notice two things: (1) the airplane is descending, and (2) the nose is flat or pointed just slightly up. It should be clear that in this situation, the angle between the oncoming air and the wing — the angle of attack — and the angle of the nose are different. Because the airplane is descending, the angle of attack is greater than the angle of the nose.

monksy 8y ago

> Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent. I think this is a good point on why Agile is so dangerous to the users. Agile allows for engineering to be micromanaged and due diligence from engineers to be overlooked. (It's not something the business people care about) Don't deliver without the safety feature? Well, you're underperforming... they're going to be let go pretty quickly.