5 ms·

I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossib

by jdkee 7y ago

I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossible.

ethbro 7y ago
They were held to the standard of "Get it done, yesterday." And will likely eventually be held to the standard of "How dare you let this happen?" as they're fired, to demonstrate how seriously Boeing takes safety.
skissane 7y ago
The programmer is told to implement a spec. They didn't write the spec. If the spec is flawed, do they even have the necessary domain knowledge and experience to identify those flaws? And, if they can identify them, and they speak up and nobody listens, what do they do next? Implement it anyway, or quit and let somebody else implement it instead? Especially if they don't even work for Boeing, but for some subcontractor.
- ethbro 7y ago
  The logic follows that if a malicious company deliberately subcontracts all their development, then they've completely indemnified themselves against all claims. While simultaneously providing no opportunity for subcontractors to bring noticed issues to their attention. Which seems a pretty bad state of affairs to trust when building things like airplanes.
  - luckydata 7y ago
    It is. that's why strong labor laws protect everyone, not just the workers, and why well run Unions are so fundamental to the sustainability of an economy.
  - throwaheyy 7y ago
    That is why high risk industries have regulators like the FAA instead of just relying on the fear of liability. In terms of liability, Boeing can try push it to the subbies and the subbies can try and push it back to Boeing. Both are trying to bamboozle the non-technical lawyers. The purpose of the FAA is to cut through that crap and enforce actual, effective change through sanctions or otherwise, and they didn’t do that. That’s what fell apart here.
    - to11mtm 7y ago
      To wit: In my state, Gambling is legalized. I remember my surprise when a friend who worked in the compliance side of the business actually knew what MD5 was (Back in 2007.) She wasn't a 'technical' person either. She explained that they actually had to audit the slot machines to make sure that the code running on them had a hash that matched a codebase that had been audited and approved by the state regulatory body. So, the practice for auditing code by a regulatory body is nothing new. If we do it for money, FFS can we do it when there are actual lives involved?
      - forgetfulusr 7y ago
        Someone weighed that the short-term money here was better than long-term. I think that quarterly earnings reports is the wrong optimization for some industries. If only the execs at Boeing could imagine each of their planes carrying suitcases of their Bonuses.
    - jdkee 7y ago
      As for the FAA, they appear to be in the throes of regulatory capture on this issue. https://www.economist.com/business/2019/03/23/regulatory-capture-may-be-responsible-for-boeings-recent-problems https://www.economist.com/business/2019/03/23/regulatory-cap...
- _ph_ 7y ago
  Which is why it is so important to keep safety-critical software in-house. Because then the developers shouldn't be limited in knowledge about their prime domain, software engineering, but also about the domain the software is used for. So real aeronautic engineers need to be part of the software teams. People who would at least have a chance to detect fatal flaws in the specs they are supposed to implement, if they were not part of the spec creation from the beginning.
jhayward 7y ago
I haven't seen any evidence that the coders were doing anything other than accurately implementing the system design that the control systems engineers dictated. What ethical safety standards would you have them implement? I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.
- salawat 7y ago
  I don't implement anything I can't/don't understand. I'm a generalist, so I make it my business to know a bit of everyone else's business. If I can't look at a spec without seeing issues down the chain that the spec makes no mention of, I end up feeling that it is my duty to make sure to raise the question until I am satisfied with the answer. I don't always get the most satisfying answer, and I haven't had to put the career on the line by doing so yet; but I'm prepared to do so nevertheless. I will not be part of the next THERAC-25/MAX fiasco. And if I've learned anything from this decade, it is that engineers as a whole may need to organize against those that would seek to have us do unethical work. It wouldn't stop the practice, and God help me, I don't want the field locked behind accreditation/licensure... However, I don't see any other defense or measure that would allow for putting the kebash on bad work. There has to be a price for bad corporate behavior in terms of ruthlessly pursuing performance that can only be met through wink wink nudge nudge style inducement to unethical behavior. At least, no way besides publically outting a company's dirty laundry. That really isn't satisfying though, because that requires a sacrifice of somebody's integrity every time, and no one wants to touch you after that. I just can't converge to a satisfying middle-ground with the right incentives. Besides maybe anonymous whistleblowing to an appropriate watchdog agency. Even then though, issues are raised in that you are leaving the regulation up to people who feel insecure reporting something when they have everything to lose. It is a frustrating issue to say the least.
  - throwaway890127 7y ago
    I've worked on systems in 3 categories: 1. Safety 2. Non-Safety 3. "Safety" For #3 I mean it's "we realize that failure has bigger repercussions than a fail-whale, but we can't afford to do any of the ISO processes that have been proven to work." Sometimes I feel like my only job on those sorts of systems is to bang the "Normalization of deviance is not okay" drum in every meeting. All failures need to go to the PM and get signed off on, otherwise the PM has a false sense of the actual reliability of the system. If the PM wants to get more budget for safety concerns, they should be able to hand a stack of 100s of pages of papers to whomever controls the purse strings and say "These are the failures in the last N days" If all they can say is "some of my engineers have expressed concerns" then 0 change will happen.
MaxfordAndSons 7y ago
Seems unlikely given that they were likely contractors: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...

by jdkee 7y ago

I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossible.

ethbro 7y ago

They were held to the standard of "Get it done, yesterday." And will likely eventually be held to the standard of "How dare you let this happen?" as they're fired, to demonstrate how seriously Boeing takes safety.

skissane 7y ago

The programmer is told to implement a spec. They didn't write the spec. If the spec is flawed, do they even have the necessary domain knowledge and experience to identify those flaws? And, if they can identify them, and they speak up and nobody listens, what do they do next? Implement it anyway, or quit and let somebody else implement it instead? Especially if they don't even work for Boeing, but for some subcontractor.

ethbro 7y ago
The logic follows that if a malicious company deliberately subcontracts all their development, then they've completely indemnified themselves against all claims. While simultaneously providing no opportunity for subcontractors to bring noticed issues to their attention. Which seems a pretty bad state of affairs to trust when building things like airplanes.
- luckydata 7y ago
  It is. that's why strong labor laws protect everyone, not just the workers, and why well run Unions are so fundamental to the sustainability of an economy.
- throwaheyy 7y ago
  That is why high risk industries have regulators like the FAA instead of just relying on the fear of liability. In terms of liability, Boeing can try push it to the subbies and the subbies can try and push it back to Boeing. Both are trying to bamboozle the non-technical lawyers. The purpose of the FAA is to cut through that crap and enforce actual, effective change through sanctions or otherwise, and they didn’t do that. That’s what fell apart here.
  - to11mtm 7y ago
    To wit: In my state, Gambling is legalized. I remember my surprise when a friend who worked in the compliance side of the business actually knew what MD5 was (Back in 2007.) She wasn't a 'technical' person either. She explained that they actually had to audit the slot machines to make sure that the code running on them had a hash that matched a codebase that had been audited and approved by the state regulatory body. So, the practice for auditing code by a regulatory body is nothing new. If we do it for money, FFS can we do it when there are actual lives involved?
    - forgetfulusr 7y ago
      Someone weighed that the short-term money here was better than long-term. I think that quarterly earnings reports is the wrong optimization for some industries. If only the execs at Boeing could imagine each of their planes carrying suitcases of their Bonuses.
  - jdkee 7y ago
    As for the FAA, they appear to be in the throes of regulatory capture on this issue. https://www.economist.com/business/2019/03/23/regulatory-capture-may-be-responsible-for-boeings-recent-problems https://www.economist.com/business/2019/03/23/regulatory-cap...
_ph_ 7y ago
Which is why it is so important to keep safety-critical software in-house. Because then the developers shouldn't be limited in knowledge about their prime domain, software engineering, but also about the domain the software is used for. So real aeronautic engineers need to be part of the software teams. People who would at least have a chance to detect fatal flaws in the specs they are supposed to implement, if they were not part of the spec creation from the beginning.

jhayward 7y ago

I haven't seen any evidence that the coders were doing anything other than accurately implementing the system design that the control systems engineers dictated. What ethical safety standards would you have them implement? I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.

salawat 7y ago
I don't implement anything I can't/don't understand. I'm a generalist, so I make it my business to know a bit of everyone else's business. If I can't look at a spec without seeing issues down the chain that the spec makes no mention of, I end up feeling that it is my duty to make sure to raise the question until I am satisfied with the answer. I don't always get the most satisfying answer, and I haven't had to put the career on the line by doing so yet; but I'm prepared to do so nevertheless. I will not be part of the next THERAC-25/MAX fiasco. And if I've learned anything from this decade, it is that engineers as a whole may need to organize against those that would seek to have us do unethical work. It wouldn't stop the practice, and God help me, I don't want the field locked behind accreditation/licensure... However, I don't see any other defense or measure that would allow for putting the kebash on bad work. There has to be a price for bad corporate behavior in terms of ruthlessly pursuing performance that can only be met through wink wink nudge nudge style inducement to unethical behavior. At least, no way besides publically outting a company's dirty laundry. That really isn't satisfying though, because that requires a sacrifice of somebody's integrity every time, and no one wants to touch you after that. I just can't converge to a satisfying middle-ground with the right incentives. Besides maybe anonymous whistleblowing to an appropriate watchdog agency. Even then though, issues are raised in that you are leaving the regulation up to people who feel insecure reporting something when they have everything to lose. It is a frustrating issue to say the least.
- throwaway890127 7y ago
  I've worked on systems in 3 categories: 1. Safety 2. Non-Safety 3. "Safety" For #3 I mean it's "we realize that failure has bigger repercussions than a fail-whale, but we can't afford to do any of the ISO processes that have been proven to work." Sometimes I feel like my only job on those sorts of systems is to bang the "Normalization of deviance is not okay" drum in every meeting. All failures need to go to the PM and get signed off on, otherwise the PM has a false sense of the actual reliability of the system. If the PM wants to get more budget for safety concerns, they should be able to hand a stack of 100s of pages of papers to whomever controls the purse strings and say "These are the failures in the last N days" If all they can say is "some of my engineers have expressed concerns" then 0 change will happen.

MaxfordAndSons 7y ago

Seems unlikely given that they were likely contractors: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...