7 ms·

Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers (2019)

97 points by unknownaccount 4y ago

allears 4y ago
Story from 2019.
- metadat 4y ago
  How has this worked out, 3 years later?
  - abudabi123 4y ago
    What are the numbers adjusted for inflation since then and how do they breakdown counting stockbuyback transactions, management and above corporate headquarter placeholder renumeration packages, engineers and below, and does the software perform fleetwide upgrade as Tesla does over the air?
deleted 4y ago
[deleted]
1letterunixname 4y ago
I would hazard a guess that principal professional software engineer would make around $350-600k USD. Let's say you need $10 million USD / year to run a fundamental embedded systems team and another $20 for specifics applications on a project. Call it $30 million. Suppose the economic damage of a $2.6 B fine, $ 1 B loss in intangible goodwill, $2 B in lost sales on top of the pandemic, and killing 346 people total say $8 B. That would fund about 5k top industry developers for about 5 years, or 26k person-years of effort. Or, 244k person-years of $9 USD/hr effort. Good luck managing the code quality and project management of such a large project where there is little investment or concern for who's writing code. TL;DR: Don't competitively incentivize short-term cost savings by strip-mining fundamental capabilities to deliver by throwing human capital to the wind as an interchangeable "fast-food worker" approach.
- kevin_thibedeau 4y ago
  Those are SV bubble salaries. Companies that make things don't pay that much.
  - kodah 4y ago
    I used to live outside of Silicon Valley, can confirm that most companies don't pay that much. However, what you didn't say is far more interesting: that it's okay that they don't. That part is the real travesty, that someone would look at a salary and say, "They don't deserve that" rather than, "I deserve that too." The fact is that software engineering isn't about writing code. That bit is quite trivial in any engineers day. What's hard, dare I say what you're paid to do, is to solve problems. Process problems, efficiency problems, resiliency problems, security problems - you're made to answer questions, or develop systems that answer questions to some of the hardest problems corporations have - and you'd ask for anything less? Boeing most certainly can afford to pay the price especially when human lives are at stake.
    - hef19898 4y ago
      No amount of SV salaries wouod have prevented the MAX and MCAS disasters, just saying.
    - kevin_thibedeau 4y ago
      In the real world, programmers and engineers are only a relatively small cohort of the people involved in making something that can be sold. There are 10x more people handling procurement, manufacturing, testing, and a host of other activities. They can't all make 350k and it doesn't stand to reason that the design side should get a disproportionate share.
  - Sevii 4y ago
    They paid billions in fines instead.
    - sofixa 4y ago
fallingmeat 4y ago
shouldn’t matter who does it or for how much, as long as the sw lifecycle process is followed (which would include supplier oversight). Actually writing code, btw, is the cheap/easy part- designing correct system and sw requirements, along with verification, is much more expensive.
- tdeck 4y ago
  I don't know, there's a certain point where the cost itself is a bad sign and software engineering for $9/hour is definitely below that point. If you found out the pilot of your flight was making $9 an hour, wouldn't you be concerned?
  - hef19898 4y ago
    What makes you think pilots, depending on region, are making aignificabtly more than that? Heck, some are even paying to get the necessary flight hours for their type ratings...
snehk 4y ago
But it wasn't a programming error, right? So why would it matter whether the code was written by an unpaid intern, a $9/h engineer or their senior software engineer? The code seemingly did exactly what it was supposed to.
- solotronics 4y ago
  My subjective opinion is that if you hire experts, not just programmers but experts at aviation programming, then possibly someone would have noticed a gap in the logic of the requirements and systems engineering. Who knows for sure? But I know plenty of contractors who write good code but oftentimes it's the architect level expert that knows what's beyond the code. Especially when the code is directly responsible for a physical system.
  - landemva 4y ago
    MCAS was implemented to cover up the flight characteristic changes which happened when big fans were mounted further forward and higher (else they would scrape on runway). MCAS attempted to make it appear to the pilot that the plane performed like other 737s. In my opinion the 737MAX needs a distinct type certification. This would dramatically raise costs, so it is covered up with software. FAA and regulators in other countries don't care. This management and regulator problem is enabled by all off the passengers who pay for seats on this plane.
    - dotancohen 4y ago
      I've said it before and I'll continue to say it despite the downvotes: Ask what plane a flight is on, and refuse the flight if it is moved to a 737 Max. I don't care if Boeing decided "this revision" of the 737 Max is safe. I insist on only flying in airplanes that the pilots were trained to fly on, without a compatibility layer between the pilots' training and the airframe characteristics.
      - pas 4y ago
        just an interesting tidbit: all 737 pilots were trained how to handle a runaway stabilizer. The crime of the 737Max was that the MCAS fucking shit into the wind and introduced an unbelievably dumb stochastic failure mode (every X seconds MCAS adjusted the stabilizer), and the pilots had no idea what's doing it.
nottorp 4y ago
Were they any good? (Although we know that by now). I don't think how much they will work for is relevant.
mikessoft_gmail 4y ago
Flight software is a difficult one. I was glad to work with one very good German engineer during summer 2020
gpraghu 4y ago
This is a shoddy article, and conflates unrelated issues. First, it wasn’t a software issue at all that grounded the plane. Second, Boeing has categorically said that MCAS
- dotancohen 4y ago
  > Boeing has categorically said that MCAS Was he posting from in-flight WIFI on a 737 Max?
rramadass 4y ago
Click-bait title/article to "whitewash" Boeing's responsibility in the disaster; Excerpt from the article: Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers. So it was probably non-critical, routine drudgery software which was outsourced.
- sofixa 4y ago
  Where are the developers of the MCAS though? IMO they, alongside any PMs, engineers etc. that made the decisions, executed them, okayed this piece of software, and then lied about it in front of the regulator should have criminal liability. As a reminder, we're talking about a piece of software that could control the plane's angle of attack, pointing the nose downwards (with the obvious risk that this poses), and which relies on a single physical sensor, a pitot tube, which is known for relatively frequent (for aviation) failures due to ice and other blockages, which is why there are two of them. If the people who made those choices didn't understand the ramifications of their actions they have no place in aviation. At the very least they should be fired, but ideally they should be brought to justice. Alongside every Boeing pilot, engineer, manager, exec who knew about this, and lied about it.
  - rramadass 4y ago
    Right; Boeing should be held 100% accountable. What i was pointing out is that a lot of articles like this one have been planted in the media subtly hinting and throwing shade on these external companies (who are innocent) thus deflecting from Boeing's responsibility. This needs to be called out every time.
  - pas 4y ago
    Can you please share more details about the lies? Wasn't the sensor an AoA vane instead of a pitot tube?
StopHammoTime 4y ago
The one thing I dislike about these articles is that it seems to paint India as a place where skilled Engineers are not from, when in reality there are a lot of talented people in India and from India. What these stories should be saying is "inadequately trained or skilled staff outsourced to for $9-an-hour". There are plenty of good engineers in India, they just cost the same amount as good engineers in other places.
- illuminati1911 4y ago
  "when in reality there are a lot of talented people in India and from India." There are skilled people from India, but rarely in India I'd assume. The market for software engineers in India is not great to put it politely so the talented people will obviously go where the money and benefits are which is US, Europe etc. And yes I'm sure there are exceptions, but probably not too many or do you know anyone actually buying software engineer services from India and paying western salaries?
hrbf 4y ago
I’ve said it before and will say it again: as long as there are no painful legal consequences, including jail time, for the responsible managers and C-suite, such behavior will not stop.
marzipanWhale 4y ago
Not related to the content, but I find it interesting that the article is using ー (Katakana-Hiragana Prolonged Sound Mark) instead of em dashes (—)
MilStdJunkie 4y ago
Boeing, like ogres, has layers, except the layers are decades of bad decisions and unforced failure. Pop quiz: name a new program that Boeing has started in the last twenty years that was successful. Acquisitions, buyouts, remnant MD programs don't count. Now, during the post-COVID era, they're somehow spending all of their[1] cash in precisely those sectors that are forecasted to never recover - ever. Congratulations. That takes work. If you want to see this poetry in motion, take a look at what happens to any company bought by Boeing. If you could weaponize this kind of value destruction we'd figure out a way to put it in a JDAM and drop it on Hainan Island. The MAX software issues are at the very tippity tip tail end of a single one of these chains of idiocy, so I feel uncomfortable just piling all the sins on MCAS. The fact they even had MCAS should have been a gigantic red flag, and would have been in anything like a functional regulatory regime. The FAA PNW offices were unofficial Boeing locations even back in 2008. [1]rapidly shrinking reserves of
Nasreddin_Hodja 4y ago
> $9-an-Hour Engineers It doesn't make sense due to low prices for everything in their country.

allears 4y ago

Story from 2019.

metadat 4y ago
How has this worked out, 3 years later?
- abudabi123 4y ago
  What are the numbers adjusted for inflation since then and how do they breakdown counting stockbuyback transactions, management and above corporate headquarter placeholder renumeration packages, engineers and below, and does the software perform fleetwide upgrade as Tesla does over the air?

deleted 4y ago

[deleted]

1letterunixname 4y ago

I would hazard a guess that principal professional software engineer would make around $350-600k USD. Let's say you need $10 million USD / year to run a fundamental embedded systems team and another $20 for specifics applications on a project. Call it $30 million. Suppose the economic damage of a $2.6 B fine, $ 1 B loss in intangible goodwill, $2 B in lost sales on top of the pandemic, and killing 346 people total say $8 B. That would fund about 5k top industry developers for about 5 years, or 26k person-years of effort. Or, 244k person-years of $9 USD/hr effort. Good luck managing the code quality and project management of such a large project where there is little investment or concern for who's writing code. TL;DR: Don't competitively incentivize short-term cost savings by strip-mining fundamental capabilities to deliver by throwing human capital to the wind as an interchangeable "fast-food worker" approach.

kevin_thibedeau 4y ago
Those are SV bubble salaries. Companies that make things don't pay that much.
- kodah 4y ago
  I used to live outside of Silicon Valley, can confirm that most companies don't pay that much. However, what you didn't say is far more interesting: that it's okay that they don't. That part is the real travesty, that someone would look at a salary and say, "They don't deserve that" rather than, "I deserve that too." The fact is that software engineering isn't about writing code. That bit is quite trivial in any engineers day. What's hard, dare I say what you're paid to do, is to solve problems. Process problems, efficiency problems, resiliency problems, security problems - you're made to answer questions, or develop systems that answer questions to some of the hardest problems corporations have - and you'd ask for anything less? Boeing most certainly can afford to pay the price especially when human lives are at stake.
  - hef19898 4y ago
    No amount of SV salaries wouod have prevented the MAX and MCAS disasters, just saying.
  - kevin_thibedeau 4y ago
    In the real world, programmers and engineers are only a relatively small cohort of the people involved in making something that can be sold. There are 10x more people handling procurement, manufacturing, testing, and a host of other activities. They can't all make 350k and it doesn't stand to reason that the design side should get a disproportionate share.
- Sevii 4y ago
  They paid billions in fines instead.
  - sofixa 4y ago

fallingmeat 4y ago

shouldn’t matter who does it or for how much, as long as the sw lifecycle process is followed (which would include supplier oversight). Actually writing code, btw, is the cheap/easy part- designing correct system and sw requirements, along with verification, is much more expensive.

tdeck 4y ago
I don't know, there's a certain point where the cost itself is a bad sign and software engineering for $9/hour is definitely below that point. If you found out the pilot of your flight was making $9 an hour, wouldn't you be concerned?
- hef19898 4y ago
  What makes you think pilots, depending on region, are making aignificabtly more than that? Heck, some are even paying to get the necessary flight hours for their type ratings...

snehk 4y ago

But it wasn't a programming error, right? So why would it matter whether the code was written by an unpaid intern, a $9/h engineer or their senior software engineer? The code seemingly did exactly what it was supposed to.

solotronics 4y ago
My subjective opinion is that if you hire experts, not just programmers but experts at aviation programming, then possibly someone would have noticed a gap in the logic of the requirements and systems engineering. Who knows for sure? But I know plenty of contractors who write good code but oftentimes it's the architect level expert that knows what's beyond the code. Especially when the code is directly responsible for a physical system.
- landemva 4y ago
  MCAS was implemented to cover up the flight characteristic changes which happened when big fans were mounted further forward and higher (else they would scrape on runway). MCAS attempted to make it appear to the pilot that the plane performed like other 737s. In my opinion the 737MAX needs a distinct type certification. This would dramatically raise costs, so it is covered up with software. FAA and regulators in other countries don't care. This management and regulator problem is enabled by all off the passengers who pay for seats on this plane.
  - dotancohen 4y ago
    I've said it before and I'll continue to say it despite the downvotes: Ask what plane a flight is on, and refuse the flight if it is moved to a 737 Max. I don't care if Boeing decided "this revision" of the 737 Max is safe. I insist on only flying in airplanes that the pilots were trained to fly on, without a compatibility layer between the pilots' training and the airframe characteristics.
    - pas 4y ago
      just an interesting tidbit: all 737 pilots were trained how to handle a runaway stabilizer. The crime of the 737Max was that the MCAS fucking shit into the wind and introduced an unbelievably dumb stochastic failure mode (every X seconds MCAS adjusted the stabilizer), and the pilots had no idea what's doing it.

nottorp 4y ago

Were they any good? (Although we know that by now). I don't think how much they will work for is relevant.

mikessoft_gmail 4y ago

Flight software is a difficult one. I was glad to work with one very good German engineer during summer 2020

gpraghu 4y ago

This is a shoddy article, and conflates unrelated issues. First, it wasn’t a software issue at all that grounded the plane. Second, Boeing has categorically said that MCAS

dotancohen 4y ago
> Boeing has categorically said that MCAS Was he posting from in-flight WIFI on a 737 Max?

rramadass 4y ago

Click-bait title/article to "whitewash" Boeing's responsibility in the disaster; Excerpt from the article: Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers. So it was probably non-critical, routine drudgery software which was outsourced.

sofixa 4y ago
Where are the developers of the MCAS though? IMO they, alongside any PMs, engineers etc. that made the decisions, executed them, okayed this piece of software, and then lied about it in front of the regulator should have criminal liability. As a reminder, we're talking about a piece of software that could control the plane's angle of attack, pointing the nose downwards (with the obvious risk that this poses), and which relies on a single physical sensor, a pitot tube, which is known for relatively frequent (for aviation) failures due to ice and other blockages, which is why there are two of them. If the people who made those choices didn't understand the ramifications of their actions they have no place in aviation. At the very least they should be fired, but ideally they should be brought to justice. Alongside every Boeing pilot, engineer, manager, exec who knew about this, and lied about it.
- rramadass 4y ago
  Right; Boeing should be held 100% accountable. What i was pointing out is that a lot of articles like this one have been planted in the media subtly hinting and throwing shade on these external companies (who are innocent) thus deflecting from Boeing's responsibility. This needs to be called out every time.
- pas 4y ago
  Can you please share more details about the lies? Wasn't the sensor an AoA vane instead of a pitot tube?

StopHammoTime 4y ago

The one thing I dislike about these articles is that it seems to paint India as a place where skilled Engineers are not from, when in reality there are a lot of talented people in India and from India. What these stories should be saying is "inadequately trained or skilled staff outsourced to for $9-an-hour". There are plenty of good engineers in India, they just cost the same amount as good engineers in other places.

illuminati1911 4y ago
"when in reality there are a lot of talented people in India and from India." There are skilled people from India, but rarely in India I'd assume. The market for software engineers in India is not great to put it politely so the talented people will obviously go where the money and benefits are which is US, Europe etc. And yes I'm sure there are exceptions, but probably not too many or do you know anyone actually buying software engineer services from India and paying western salaries?

hrbf 4y ago

I’ve said it before and will say it again: as long as there are no painful legal consequences, including jail time, for the responsible managers and C-suite, such behavior will not stop.

marzipanWhale 4y ago

Not related to the content, but I find it interesting that the article is using ー (Katakana-Hiragana Prolonged Sound Mark) instead of em dashes (—)

MilStdJunkie 4y ago

Boeing, like ogres, has layers, except the layers are decades of bad decisions and unforced failure. Pop quiz: name a new program that Boeing has started in the last twenty years that was successful. Acquisitions, buyouts, remnant MD programs don't count. Now, during the post-COVID era, they're somehow spending all of their[1] cash in precisely those sectors that are forecasted to never recover - ever. Congratulations. That takes work. If you want to see this poetry in motion, take a look at what happens to any company bought by Boeing. If you could weaponize this kind of value destruction we'd figure out a way to put it in a JDAM and drop it on Hainan Island. The MAX software issues are at the very tippity tip tail end of a single one of these chains of idiocy, so I feel uncomfortable just piling all the sins on MCAS. The fact they even had MCAS should have been a gigantic red flag, and would have been in anything like a functional regulatory regime. The FAA PNW offices were unofficial Boeing locations even back in 2008. [1]rapidly shrinking reserves of

Nasreddin_Hodja 4y ago

> $9-an-Hour Engineers It doesn't make sense due to low prices for everything in their country.