6 ms·

The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little. The main issue wa

The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little. The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed. The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall. For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunities-overseas/vacancies-in-india.html https://www.airbus.com/careers/search-and-apply/opportunitie... Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.

pxue 7y ago
Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?
- ganeshkrishnan 7y ago
  https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidents https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs
  - gotdevops 7y ago
    It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22
gotdevops 7y ago
You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22
- dang 7y ago
  Could you please not cross into personal attack? and also please not post in the flamewar style? These are things we're trying to avoid on HN. The site guidelines explain, and if you'd read them and stick to the rules, we'd be grateful: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
- danielg6 7y ago
  I do think it’s weird that the parent post started out by saying the sensors were designed to spec, but is he wrong? The article even said as much. (Not MCAS though.) I didn’t read it as an excuse. It seems like you’re saying that outsourced devs should have as much accountability as home-based devs or the entity Boeing itself. I don’t agree with this. The salary does not reflect that. The work that’s assigned does not reflect that. Anecdotally working with outsourced devs and based from the comments here, no one honestly thinks that. It seems like you were triggered by the “excuses” part of the post. But what about the rest of it? He made a valid point (which I think you proved) about people using the article to discriminate. I guess I’m just wondering why you’re making generalizations. I’ve worked with many devs (who weren’t outsourced) who blame everyone else but themselves. I learned to use ‘git blame’ very early on.
  - gotdevops 7y ago
    It’s a very interesting point. I’ve worked with some very skilled devs from HCL. I’ve also seen them turn over to other firms because of salary and other firm related issues. I don’t think that piece has anything or very little to do with the fact that they are Indian. However the organizational culture is definitely flawed and it does appear to be more prevalent in large fixed bid outsourcing arrangements. I’ve witnessed it first hand being brought in to resolve issues created by the outsourcer on projects as large as 1/2bn. They difference between those projects is the mission. Those project were critical on a financial basis. This was and aircraft with peoples life’s on the line. No one dev or otherwise can say they didn’t know this code was going on an airplane. No matter their salary.
    - 7y ago
AlexCoventry 7y ago
> Articles like this are just drummed up to stir negative emotions about "others" The negativity seems more targeted at the leaders who arranged this than at the developers. Both the examples you cite are from the same comment, and even that comment is more about resentment towards management corruption than about the incompetence of the developers hired as a result of the corruption. > The software did exactly as it's supposed to do I gather new issues with the software have come to light since the accidents, as a result of the heightened scrutiny, and they seem likely to keep the planes grounded for longer than Boeing hoped.
- danielg6 7y ago
  See the post below yours lol.
  - AlexCoventry 7y ago
    Not sure which one you were referring to. The ordering of comments changes over time. Could you post a URL or identifying substring?
    - danielg6 7y ago
      That’s the joke... You can scroll up or down and find such a comment.
deleted 7y ago
[deleted]
nobodyandproud 7y ago
The problem is that developers also need to understand what they are coding, and fight back when they notice that the design is fundamentally flawed. In this case, the lack of redundancy should have been a red-flagged. This is why domain knowledge is at least as important as coding skills.
- durnygbur 7y ago
  In highly process-based environments, which outsourcing centers are, fighting back gets you harrased by the zealot managers out of the workplace.
- ganeshkrishnan 7y ago
  There were multiple redundancy built in. Multiple sensors, voting mechanism, manual override. Many of which meant re-training the pilots and Boeing wanted to avoid it. I think southwest airlines had multiple sensors
  - nobodyandproud 7y ago
    The 737 Max aircrafts that crashed relied on a single AOA sensor. A second AOA sensor is an option. Which means code was written to account for both configurations. Many, many things must happen and change to avoid allowing management to make decisions unchecked. Because this is a case of bargain-bin engineering and tunnel-visioned management ruining a once-reputable company.
  - deleted 7y ago
    [deleted]
dTal 7y ago
This comment is glib and misinformed. Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that. And your description of the MCAS specification is vastly over-simplified - it is not an autopilot and its job is to gradually change the behaviour of the plane in different flight modes (which it must detect), not to directly fly the plane as you suggest (and even two AOA sensors would be insufficient for the type of functionality you describe). The behaviour of the MCAS is absolutely in question, in particular the way it kept "re-adding" the correction, with cumulative effect, ending up fully pitched down. Secondly, if by "the doomed Air France incident" you mean AF447, your description of it is flat-out wrong. The pilot, not software, stalled and crashed the plane by manually holding the stick back for 15 minutes. The pitot tubes briefly iced over (they didn't "fail") which is not an uncommon occurrence, and the only consequence of this was that the software guardrails that would have prevented the pilot from crashing the plane were removed. All instrumentation systems were fully operational at the time of the crash.
- ganeshkrishnan 7y ago
  It kept re-adding because the flap of the aoa sensor was stuck in the "aircraft is pitching up" position. Boeing management did not add a second sensor to some planes and also pilots were not aware of how to switch off mcas. If you worked in large scale project you would know that most devs are assigned components to work on and usually have no idea what other components are or how they will interact.
  - gotdevops 7y ago
    This is where testing comes in and from the fact packet it appears that the testing was also outsourced to another Indian firm?
- sokoloff 7y ago
  Agree with everything you wrote, with the possible exception of referring to the stick-holder in the AF447 crash as a "pilot".
- mannykannot 7y ago
TarlanT 7y ago
Experienced software engineer would NEVER code something like that. Both physically and logically it makes no sense to get same AOA reading after many iterations of trimming. That clearly means something is wrong. And probably the sanity of data being fed is in question. It’s the dumbest mistake in critical system I’ve ever heard. Even in non critical software these types of logical mistakes are rare.
- mannykannot 7y ago
  AofA is a function of a number of factors, such as control inputs -- data not available to MCAS. It would be disturbing if the coders on aircraft control systems were modifying the specifications on the fly, and on the basis of misconceptions like that displayed here. Responsibility for the dangerous capabilities of MCAS lies with Boeing, which chose to fix an aerodynamic problem this way, and the FAA, which endorsed it.

pxue 7y ago

Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?

ganeshkrishnan 7y ago
https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidents https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs
- gotdevops 7y ago
  It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22

gotdevops 7y ago

You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22

dang 7y ago
Could you please not cross into personal attack? and also please not post in the flamewar style? These are things we're trying to avoid on HN. The site guidelines explain, and if you'd read them and stick to the rules, we'd be grateful: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
danielg6 7y ago
I do think it’s weird that the parent post started out by saying the sensors were designed to spec, but is he wrong? The article even said as much. (Not MCAS though.) I didn’t read it as an excuse. It seems like you’re saying that outsourced devs should have as much accountability as home-based devs or the entity Boeing itself. I don’t agree with this. The salary does not reflect that. The work that’s assigned does not reflect that. Anecdotally working with outsourced devs and based from the comments here, no one honestly thinks that. It seems like you were triggered by the “excuses” part of the post. But what about the rest of it? He made a valid point (which I think you proved) about people using the article to discriminate. I guess I’m just wondering why you’re making generalizations. I’ve worked with many devs (who weren’t outsourced) who blame everyone else but themselves. I learned to use ‘git blame’ very early on.
- gotdevops 7y ago
  It’s a very interesting point. I’ve worked with some very skilled devs from HCL. I’ve also seen them turn over to other firms because of salary and other firm related issues. I don’t think that piece has anything or very little to do with the fact that they are Indian. However the organizational culture is definitely flawed and it does appear to be more prevalent in large fixed bid outsourcing arrangements. I’ve witnessed it first hand being brought in to resolve issues created by the outsourcer on projects as large as 1/2bn. They difference between those projects is the mission. Those project were critical on a financial basis. This was and aircraft with peoples life’s on the line. No one dev or otherwise can say they didn’t know this code was going on an airplane. No matter their salary.
  - 7y ago

AlexCoventry 7y ago

> Articles like this are just drummed up to stir negative emotions about "others" The negativity seems more targeted at the leaders who arranged this than at the developers. Both the examples you cite are from the same comment, and even that comment is more about resentment towards management corruption than about the incompetence of the developers hired as a result of the corruption. > The software did exactly as it's supposed to do I gather new issues with the software have come to light since the accidents, as a result of the heightened scrutiny, and they seem likely to keep the planes grounded for longer than Boeing hoped.

danielg6 7y ago
See the post below yours lol.
- AlexCoventry 7y ago
  Not sure which one you were referring to. The ordering of comments changes over time. Could you post a URL or identifying substring?
  - danielg6 7y ago
    That’s the joke... You can scroll up or down and find such a comment.

deleted 7y ago

[deleted]

nobodyandproud 7y ago

The problem is that developers also need to understand what they are coding, and fight back when they notice that the design is fundamentally flawed. In this case, the lack of redundancy should have been a red-flagged. This is why domain knowledge is at least as important as coding skills.

durnygbur 7y ago
In highly process-based environments, which outsourcing centers are, fighting back gets you harrased by the zealot managers out of the workplace.
ganeshkrishnan 7y ago
There were multiple redundancy built in. Multiple sensors, voting mechanism, manual override. Many of which meant re-training the pilots and Boeing wanted to avoid it. I think southwest airlines had multiple sensors
- nobodyandproud 7y ago
  The 737 Max aircrafts that crashed relied on a single AOA sensor. A second AOA sensor is an option. Which means code was written to account for both configurations. Many, many things must happen and change to avoid allowing management to make decisions unchecked. Because this is a case of bargain-bin engineering and tunnel-visioned management ruining a once-reputable company.
- deleted 7y ago
  [deleted]

dTal 7y ago

This comment is glib and misinformed. Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that. And your description of the MCAS specification is vastly over-simplified - it is not an autopilot and its job is to gradually change the behaviour of the plane in different flight modes (which it must detect), not to directly fly the plane as you suggest (and even two AOA sensors would be insufficient for the type of functionality you describe). The behaviour of the MCAS is absolutely in question, in particular the way it kept "re-adding" the correction, with cumulative effect, ending up fully pitched down. Secondly, if by "the doomed Air France incident" you mean AF447, your description of it is flat-out wrong. The pilot, not software, stalled and crashed the plane by manually holding the stick back for 15 minutes. The pitot tubes briefly iced over (they didn't "fail") which is not an uncommon occurrence, and the only consequence of this was that the software guardrails that would have prevented the pilot from crashing the plane were removed. All instrumentation systems were fully operational at the time of the crash.

ganeshkrishnan 7y ago
It kept re-adding because the flap of the aoa sensor was stuck in the "aircraft is pitching up" position. Boeing management did not add a second sensor to some planes and also pilots were not aware of how to switch off mcas. If you worked in large scale project you would know that most devs are assigned components to work on and usually have no idea what other components are or how they will interact.
- gotdevops 7y ago
  This is where testing comes in and from the fact packet it appears that the testing was also outsourced to another Indian firm?
sokoloff 7y ago
Agree with everything you wrote, with the possible exception of referring to the stick-holder in the AF447 crash as a "pilot".
mannykannot 7y ago

TarlanT 7y ago

Experienced software engineer would NEVER code something like that. Both physically and logically it makes no sense to get same AOA reading after many iterations of trimming. That clearly means something is wrong. And probably the sanity of data being fed is in question. It’s the dumbest mistake in critical system I’ve ever heard. Even in non critical software these types of logical mistakes are rare.

mannykannot 7y ago
AofA is a function of a number of factors, such as control inputs -- data not available to MCAS. It would be disturbing if the coders on aircraft control systems were modifying the specifications on the fly, and on the basis of misconceptions like that displayed here. Responsibility for the dangerous capabilities of MCAS lies with Boeing, which chose to fix an aerodynamic problem this way, and the FAA, which endorsed it.