5 ms·

The MCAS is a good example, why certain software must not be outsourced. Implementing the software correctly to spec isn't enough, you need enough understanding

by _ph_ 7y ago

The MCAS is a good example, why certain software must not be outsourced. Implementing the software correctly to spec isn't enough, you need enough understanding of the application to question the spec itself.

bdavis__ 7y ago
anything that is connected to the flight controls, is too important to sub out (unless you are subbing it out to someone better that you are). making an airplane is a huge endeavor that crosses across many domains. you have to have competency up and down the 'stack'. Boeing does. Maybe this will cause a re-look at outsourcing policies. (edit. yes, this software was not outsourced. but it may add another dimension other than cost for future decisions)
- prasooncc 7y ago
  It has been mentioned in the article that MCAS was never outsourced. Also it’s unfair to blame the software engineers who are building it to specification. It would be fantastical to assume that Boeing management( the same one ploughing through engineering with bean counting) would give a rats crap about a low level engineer raising a concern.
  - _ph_ 7y ago
    It has been mentioned in the article that MCAS was never outsourced. Where did the article say that? To me it explicitly claims it was outsourced: the McDonnell Douglas influenced Boeing new one tried to patch the problem with software. And it was bad software, written by to engineers paid $9/dollar an hour. And I am not blaming the software engineers for building it to the specification. In contrary, I clearly stated, that just building it to specification isn't enough for such critical software. Such software needs to be developed in teams where enough team members are senior enough to properly raise concerns where appropriate.
    - glitchdout 7y ago
      Nowhere in the [original reporting][1] does it say that MCAS was outsourced. This article (The Coming Boeing Bailout?) wrongly asserts that MCAS was outsourced. There's no corroboration for that. Mind you, I'm not on Boeing's side. I'm just clarifying what's been reported so far. [1]: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
      - _ph_ 7y ago
        I had understood the part of the article, which I quoted as that MCAS was outsourced. If it wasn't then the article is to blame and my point is moot.
      - michaelmrose 7y ago
        https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... >In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max. >The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”
        glitchdout 7y ago
        Nowhere in that quote does it say that: 1. HCL wrote the code for MCAS; or 2. HCL wrote all the code of the Max. But again, I'm not defending Boeing. I think they made some (deadly) mistakes. But so far there's no proof that the MCAS software was outsourced.
  - puzzledobserver 7y ago
    Not sure why this comment is being downvoted. First, none of the software in question was outsourced to the company. Second, the Bloomberg article does not describe the problems Boeing had with the software engineers from HCL, beyond a cryptic, "it took many rounds going back and forth because the code was not done correctly." Was it a genuine misunderstanding of ambiguous language? Was it because the programmers did not have a background in the aerospace problem domain? Or was it because the engineers had a poor grasp of the vocabulary? It seems improper to assume the third situation in the absence of any additional evidence. Finally, I am not sure what part of "software being outsourced" is considered outrageous. Is there evidence that a software engineer employed by Boeing, rather than by company XYZ, is less likely to produce faulty code? Or is more likely to understand the complex downstream effects of their actions and the equally complex upstream design decisions which informed them? Also, countries don't write code. Companies don't write code. People do. That "India doesn't have a deep background in aerospace" doesn't mean much at the level of individual engineers. I am skeptical that the most highly skilled American programmer has a deep background in aerospace either. Sorry about the rant: something about the Blookmberg article came across as xenophobic innuendo.
    - senderista 7y ago
      MSFT had a similar experience with HCL (I think it was them, one of the large Indian outsourcing firms in any case) when I was working on Vista. All fixes for bugs identified by static analyzers (PREfix/PREfast) were outsourced to contract programmers who had zero familiarity with the Windows source code. The “fixes” they sent back almost invariably introduced new bugs, and the whole back-and-forth process took much more time and effort (and presumably money) than just having the code owners fix the bugs in the first place.
partha42 7y ago
Software engineers are not aerospace engineers. I doubt the software people would have understood the aerodynamic effects of the MCAS system by looking at the requirements.
- bdavis__ 7y ago
  some would, some would not. those that have been doing aircraft software for 20 years probably have a fair knowledge of airplanes, aerodynamics, and flight controls. your comment expresses my feelings exactly on 'was this a software failure?'. No. But software was involved.
- _ph_ 7y ago
  That is the problem with having pure software engineers implementing such a software. They need to have some aerospace engineering knowledge too. Or at least work directly in one team with aerospace engineers. I am working as a "software engineer" in the semiconductor industry. By education, I am a physicist. I am in my job for more than 10 years now, and beyond my university education, I have quite a grasp of electrical engineering. Furthermore, most of my colleagues are eletrical engineers. So in doubt, I can just walk over to their desk and discuss any question beyond pure software development issues. And some of the software developers here are just electrical engineers, which expanded their skill profile into software territory. Doing business this way is quite more expensive than having software developed by a contractor, but it creates safeguards.
  - sn41 7y ago
    That is true. But please keep in mind that the whole MCAS fiasco was precipitated by a dubious cost-cutting initiative related to an engine placed more in front than would be aerodynamically stable. The game was lost in the first quarter. The whole point is that civilian aircraft, unlike fighter jets, have to be aerodynamically stable without the need for such software, whether developed by holistic engineers, or compartmentalised subcontractors.
    - _ph_ 7y ago
senderista 7y ago
That assertion in the article seems to be incorrect according to the linked Bloomberg article, which states that the MCAS software was not outsourced: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-.... Given that apparent factual error, I’m suspicious of the other claims, which are highly rhetorical and mostly asserted without evidence.

by _ph_ 7y ago

bdavis__ 7y ago

anything that is connected to the flight controls, is too important to sub out (unless you are subbing it out to someone better that you are). making an airplane is a huge endeavor that crosses across many domains. you have to have competency up and down the 'stack'. Boeing does. Maybe this will cause a re-look at outsourcing policies. (edit. yes, this software was not outsourced. but it may add another dimension other than cost for future decisions)

prasooncc 7y ago
It has been mentioned in the article that MCAS was never outsourced. Also it’s unfair to blame the software engineers who are building it to specification. It would be fantastical to assume that Boeing management( the same one ploughing through engineering with bean counting) would give a rats crap about a low level engineer raising a concern.
- _ph_ 7y ago
  It has been mentioned in the article that MCAS was never outsourced. Where did the article say that? To me it explicitly claims it was outsourced: the McDonnell Douglas influenced Boeing new one tried to patch the problem with software. And it was bad software, written by to engineers paid $9/dollar an hour. And I am not blaming the software engineers for building it to the specification. In contrary, I clearly stated, that just building it to specification isn't enough for such critical software. Such software needs to be developed in teams where enough team members are senior enough to properly raise concerns where appropriate.
  - glitchdout 7y ago
    Nowhere in the [original reporting][1] does it say that MCAS was outsourced. This article (The Coming Boeing Bailout?) wrongly asserts that MCAS was outsourced. There's no corroboration for that. Mind you, I'm not on Boeing's side. I'm just clarifying what's been reported so far. [1]: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
    - _ph_ 7y ago
      I had understood the part of the article, which I quoted as that MCAS was outsourced. If it wasn't then the article is to blame and my point is moot.
    - michaelmrose 7y ago
      https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... >In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max. >The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”
      - glitchdout 7y ago
        Nowhere in that quote does it say that: 1. HCL wrote the code for MCAS; or 2. HCL wrote all the code of the Max. But again, I'm not defending Boeing. I think they made some (deadly) mistakes. But so far there's no proof that the MCAS software was outsourced.
- puzzledobserver 7y ago
  Not sure why this comment is being downvoted. First, none of the software in question was outsourced to the company. Second, the Bloomberg article does not describe the problems Boeing had with the software engineers from HCL, beyond a cryptic, "it took many rounds going back and forth because the code was not done correctly." Was it a genuine misunderstanding of ambiguous language? Was it because the programmers did not have a background in the aerospace problem domain? Or was it because the engineers had a poor grasp of the vocabulary? It seems improper to assume the third situation in the absence of any additional evidence. Finally, I am not sure what part of "software being outsourced" is considered outrageous. Is there evidence that a software engineer employed by Boeing, rather than by company XYZ, is less likely to produce faulty code? Or is more likely to understand the complex downstream effects of their actions and the equally complex upstream design decisions which informed them? Also, countries don't write code. Companies don't write code. People do. That "India doesn't have a deep background in aerospace" doesn't mean much at the level of individual engineers. I am skeptical that the most highly skilled American programmer has a deep background in aerospace either. Sorry about the rant: something about the Blookmberg article came across as xenophobic innuendo.
  - senderista 7y ago
    MSFT had a similar experience with HCL (I think it was them, one of the large Indian outsourcing firms in any case) when I was working on Vista. All fixes for bugs identified by static analyzers (PREfix/PREfast) were outsourced to contract programmers who had zero familiarity with the Windows source code. The “fixes” they sent back almost invariably introduced new bugs, and the whole back-and-forth process took much more time and effort (and presumably money) than just having the code owners fix the bugs in the first place.

partha42 7y ago

Software engineers are not aerospace engineers. I doubt the software people would have understood the aerodynamic effects of the MCAS system by looking at the requirements.

bdavis__ 7y ago
some would, some would not. those that have been doing aircraft software for 20 years probably have a fair knowledge of airplanes, aerodynamics, and flight controls. your comment expresses my feelings exactly on 'was this a software failure?'. No. But software was involved.
_ph_ 7y ago
That is the problem with having pure software engineers implementing such a software. They need to have some aerospace engineering knowledge too. Or at least work directly in one team with aerospace engineers. I am working as a "software engineer" in the semiconductor industry. By education, I am a physicist. I am in my job for more than 10 years now, and beyond my university education, I have quite a grasp of electrical engineering. Furthermore, most of my colleagues are eletrical engineers. So in doubt, I can just walk over to their desk and discuss any question beyond pure software development issues. And some of the software developers here are just electrical engineers, which expanded their skill profile into software territory. Doing business this way is quite more expensive than having software developed by a contractor, but it creates safeguards.
- sn41 7y ago
  That is true. But please keep in mind that the whole MCAS fiasco was precipitated by a dubious cost-cutting initiative related to an engine placed more in front than would be aerodynamically stable. The game was lost in the first quarter. The whole point is that civilian aircraft, unlike fighter jets, have to be aerodynamically stable without the need for such software, whether developed by holistic engineers, or compartmentalised subcontractors.
  - _ph_ 7y ago

senderista 7y ago

That assertion in the article seems to be incorrect according to the linked Bloomberg article, which states that the MCAS software was not outsourced: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-.... Given that apparent factual error, I’m suspicious of the other claims, which are highly rhetorical and mostly asserted without evidence.