5 ms·

During my civil engineering studies there was an emphasis on ethics throughout classes, even the more specialized, technical ones. This is one huge difference I

During my civil engineering studies there was an emphasis on ethics throughout classes, even the more specialized, technical ones. This is one huge difference I've noticed compared to the software engineering field. There aren't these governing bodies such as ASCE or NSPE with these ethics canons. I suspect we'll see that change in the next several years.

jdkee 7y ago
I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossible.
- ethbro 7y ago
  They were held to the standard of "Get it done, yesterday." And will likely eventually be held to the standard of "How dare you let this happen?" as they're fired, to demonstrate how seriously Boeing takes safety.
- skissane 7y ago
  The programmer is told to implement a spec. They didn't write the spec. If the spec is flawed, do they even have the necessary domain knowledge and experience to identify those flaws? And, if they can identify them, and they speak up and nobody listens, what do they do next? Implement it anyway, or quit and let somebody else implement it instead? Especially if they don't even work for Boeing, but for some subcontractor.
  - ethbro 7y ago
    The logic follows that if a malicious company deliberately subcontracts all their development, then they've completely indemnified themselves against all claims. While simultaneously providing no opportunity for subcontractors to bring noticed issues to their attention. Which seems a pretty bad state of affairs to trust when building things like airplanes.
    - luckydata 7y ago
      It is. that's why strong labor laws protect everyone, not just the workers, and why well run Unions are so fundamental to the sustainability of an economy.
    - throwaheyy 7y ago
      That is why high risk industries have regulators like the FAA instead of just relying on the fear of liability. In terms of liability, Boeing can try push it to the subbies and the subbies can try and push it back to Boeing. Both are trying to bamboozle the non-technical lawyers. The purpose of the FAA is to cut through that crap and enforce actual, effective change through sanctions or otherwise, and they didn’t do that. That’s what fell apart here.
      - to11mtm 7y ago
        To wit: In my state, Gambling is legalized. I remember my surprise when a friend who worked in the compliance side of the business actually knew what MD5 was (Back in 2007.) She wasn't a 'technical' person either. She explained that they actually had to audit the slot machines to make sure that the code running on them had a hash that matched a codebase that had been audited and approved by the state regulatory body. So, the practice for auditing code by a regulatory body is nothing new. If we do it for money, FFS can we do it when there are actual lives involved?
        forgetfulusr 7y ago
        Someone weighed that the short-term money here was better than long-term. I think that quarterly earnings reports is the wrong optimization for some industries. If only the execs at Boeing could imagine each of their planes carrying suitcases of their Bonuses.
      - jdkee 7y ago
        As for the FAA, they appear to be in the throes of regulatory capture on this issue. https://www.economist.com/business/2019/03/23/regulatory-capture-may-be-responsible-for-boeings-recent-problems https://www.economist.com/business/2019/03/23/regulatory-cap...
  - _ph_ 7y ago
    Which is why it is so important to keep safety-critical software in-house. Because then the developers shouldn't be limited in knowledge about their prime domain, software engineering, but also about the domain the software is used for. So real aeronautic engineers need to be part of the software teams. People who would at least have a chance to detect fatal flaws in the specs they are supposed to implement, if they were not part of the spec creation from the beginning.
- jhayward 7y ago
  I haven't seen any evidence that the coders were doing anything other than accurately implementing the system design that the control systems engineers dictated. What ethical safety standards would you have them implement? I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.
  - salawat 7y ago
    I don't implement anything I can't/don't understand. I'm a generalist, so I make it my business to know a bit of everyone else's business. If I can't look at a spec without seeing issues down the chain that the spec makes no mention of, I end up feeling that it is my duty to make sure to raise the question until I am satisfied with the answer. I don't always get the most satisfying answer, and I haven't had to put the career on the line by doing so yet; but I'm prepared to do so nevertheless. I will not be part of the next THERAC-25/MAX fiasco. And if I've learned anything from this decade, it is that engineers as a whole may need to organize against those that would seek to have us do unethical work. It wouldn't stop the practice, and God help me, I don't want the field locked behind accreditation/licensure... However, I don't see any other defense or measure that would allow for putting the kebash on bad work. There has to be a price for bad corporate behavior in terms of ruthlessly pursuing performance that can only be met through wink wink nudge nudge style inducement to unethical behavior. At least, no way besides publically outting a company's dirty laundry. That really isn't satisfying though, because that requires a sacrifice of somebody's integrity every time, and no one wants to touch you after that. I just can't converge to a satisfying middle-ground with the right incentives. Besides maybe anonymous whistleblowing to an appropriate watchdog agency. Even then though, issues are raised in that you are leaving the regulation up to people who feel insecure reporting something when they have everything to lose. It is a frustrating issue to say the least.
    - throwaway890127 7y ago
      I've worked on systems in 3 categories: 1. Safety 2. Non-Safety 3. "Safety" For #3 I mean it's "we realize that failure has bigger repercussions than a fail-whale, but we can't afford to do any of the ISO processes that have been proven to work." Sometimes I feel like my only job on those sorts of systems is to bang the "Normalization of deviance is not okay" drum in every meeting. All failures need to go to the PM and get signed off on, otherwise the PM has a false sense of the actual reliability of the system. If the PM wants to get more budget for safety concerns, they should be able to hand a stack of 100s of pages of papers to whomever controls the purse strings and say "These are the failures in the last N days" If all they can say is "some of my engineers have expressed concerns" then 0 change will happen.
- MaxfordAndSons 7y ago
  Seems unlikely given that they were likely contractors: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
cheez 7y ago
I took ethics courses as part of comp sci. Definitely left an imprint on me.
joe_the_user 7y ago
The reason the engineers are given ethical training is because engineers (like accountants, appraisers or similar people) have historically had an institutional independence from their employers. A civil engineer in particular can choose and will not to sign off on a given project that they know will be unsafe. Given that engineers are certified by their profession, their employment situation is not precarious and so they can reasonable make this refusal (this is degrading like all professions naturally but still). Programmers have no such institutional support. If a programmer refuses a job, it goes to someone else that's it. Programmers may have ethics but ethical training a la engineers isn't going to give them any leverage for choices.
- luckydata 7y ago
  Exactly! You seem to get it, the issue is not the lack of knowledge of ethics, but the lack of power. That's one of the many, many reasons why strong unions are so important and in this country we see the effects of not having them. Without protection for people refusing to do bad things, you create a system when there's always someone desperate, hungry or unethical enough to do things that shouldn't be done.
- CobrastanJorji 7y ago
  This sounds reasonable, but I don't quite understand that last bit. If a civil engineer refuses to sign off on a project, won't the same thing happen? Why won't the job go to the next civil engineer down the line?
  - azernik 7y ago
    It's a collective action problem - the company can't fire the first engineer if they know everyone else they hire would also refuse orders. But each individual engineer has an incentive (not getting fired) to break with the group's strategy. This collective action problem is solved by coordination, through the means of the licensing body. That body can impose severe penalties (not just firing you from your current job, but from all future jobs) for anyone who betrays the group strategy, so an individual engineer can feel some more safety refusing orders in the knowledge that the whole profession will back them up. EDIT: In civil engineering, this system is propped up by the state, which requires plans to be signed off by a licensed engineer. The guild functions in this capacity as a subcontractor of the state, taking on a regulatory burden and allowing rather more severe punishments (barring someone from a profession) than would be acceptable from a purely state organ. In software, this could be enforced by similar means for safety-critical applications - the ACM, for example, could be required to license any software engineer, with the understanding that they would revoke licenses for negligence or malfeasance that didn't rise to the level or criminal liability.
jimbob45 7y ago
You don't need to take an ethics class to see that cutting corners when human lives are involved is a bad idea.
- tbrownaw 7y ago
  I think I heard somewhere that the entire purpose of Engineering as a whole, is to know which corners are safe to cut. "Anyone can build a bridge that stays up. Only an engineer can build a bridge that barely stays up." (I mean, it's obviously exaggerated for effect, but still.)
isthis1984 7y ago
Membership of the ACM has a code of ethics https://www.acm.org/code-of-ethics https://www.acm.org/code-of-ethics
aiisjustanif 7y ago
I don’t know if ASCE or NSPE are governing bodies as much they are societies. I think the Core of Engineers has way more power there from my experience in Civil. For Software that would be Homeland, FBI, and even the FAA for governing bodies specifically. But software doesn’t have ethically governing bodies because we have enough laws to cover a lot of mandatory auditing. For example: HIPAA - Medical Data Protection & Compliance Gramm-Leach-Bliley Act - Financial Data Protection & Compliance FISMA - Federal Data Compliance GDPR - PII Data Compliance PCI DSS - Credit Card Data Compliance Specific example of use: Code that is deployed that involves PCI data being handled requires a code review. There are heavy penalties for lost of these things, and I promise there are many companies focusing on this. But it could be very much improved with a proper governing body.

jdkee 7y ago

I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossible.

ethbro 7y ago
They were held to the standard of "Get it done, yesterday." And will likely eventually be held to the standard of "How dare you let this happen?" as they're fired, to demonstrate how seriously Boeing takes safety.
skissane 7y ago
The programmer is told to implement a spec. They didn't write the spec. If the spec is flawed, do they even have the necessary domain knowledge and experience to identify those flaws? And, if they can identify them, and they speak up and nobody listens, what do they do next? Implement it anyway, or quit and let somebody else implement it instead? Especially if they don't even work for Boeing, but for some subcontractor.
- ethbro 7y ago
  The logic follows that if a malicious company deliberately subcontracts all their development, then they've completely indemnified themselves against all claims. While simultaneously providing no opportunity for subcontractors to bring noticed issues to their attention. Which seems a pretty bad state of affairs to trust when building things like airplanes.
  - luckydata 7y ago
    It is. that's why strong labor laws protect everyone, not just the workers, and why well run Unions are so fundamental to the sustainability of an economy.
  - throwaheyy 7y ago
    That is why high risk industries have regulators like the FAA instead of just relying on the fear of liability. In terms of liability, Boeing can try push it to the subbies and the subbies can try and push it back to Boeing. Both are trying to bamboozle the non-technical lawyers. The purpose of the FAA is to cut through that crap and enforce actual, effective change through sanctions or otherwise, and they didn’t do that. That’s what fell apart here.
    - to11mtm 7y ago
      To wit: In my state, Gambling is legalized. I remember my surprise when a friend who worked in the compliance side of the business actually knew what MD5 was (Back in 2007.) She wasn't a 'technical' person either. She explained that they actually had to audit the slot machines to make sure that the code running on them had a hash that matched a codebase that had been audited and approved by the state regulatory body. So, the practice for auditing code by a regulatory body is nothing new. If we do it for money, FFS can we do it when there are actual lives involved?
      - forgetfulusr 7y ago
        Someone weighed that the short-term money here was better than long-term. I think that quarterly earnings reports is the wrong optimization for some industries. If only the execs at Boeing could imagine each of their planes carrying suitcases of their Bonuses.
    - jdkee 7y ago
      As for the FAA, they appear to be in the throes of regulatory capture on this issue. https://www.economist.com/business/2019/03/23/regulatory-capture-may-be-responsible-for-boeings-recent-problems https://www.economist.com/business/2019/03/23/regulatory-cap...
- _ph_ 7y ago
  Which is why it is so important to keep safety-critical software in-house. Because then the developers shouldn't be limited in knowledge about their prime domain, software engineering, but also about the domain the software is used for. So real aeronautic engineers need to be part of the software teams. People who would at least have a chance to detect fatal flaws in the specs they are supposed to implement, if they were not part of the spec creation from the beginning.
jhayward 7y ago
I haven't seen any evidence that the coders were doing anything other than accurately implementing the system design that the control systems engineers dictated. What ethical safety standards would you have them implement? I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.
- salawat 7y ago
  I don't implement anything I can't/don't understand. I'm a generalist, so I make it my business to know a bit of everyone else's business. If I can't look at a spec without seeing issues down the chain that the spec makes no mention of, I end up feeling that it is my duty to make sure to raise the question until I am satisfied with the answer. I don't always get the most satisfying answer, and I haven't had to put the career on the line by doing so yet; but I'm prepared to do so nevertheless. I will not be part of the next THERAC-25/MAX fiasco. And if I've learned anything from this decade, it is that engineers as a whole may need to organize against those that would seek to have us do unethical work. It wouldn't stop the practice, and God help me, I don't want the field locked behind accreditation/licensure... However, I don't see any other defense or measure that would allow for putting the kebash on bad work. There has to be a price for bad corporate behavior in terms of ruthlessly pursuing performance that can only be met through wink wink nudge nudge style inducement to unethical behavior. At least, no way besides publically outting a company's dirty laundry. That really isn't satisfying though, because that requires a sacrifice of somebody's integrity every time, and no one wants to touch you after that. I just can't converge to a satisfying middle-ground with the right incentives. Besides maybe anonymous whistleblowing to an appropriate watchdog agency. Even then though, issues are raised in that you are leaving the regulation up to people who feel insecure reporting something when they have everything to lose. It is a frustrating issue to say the least.
  - throwaway890127 7y ago
    I've worked on systems in 3 categories: 1. Safety 2. Non-Safety 3. "Safety" For #3 I mean it's "we realize that failure has bigger repercussions than a fail-whale, but we can't afford to do any of the ISO processes that have been proven to work." Sometimes I feel like my only job on those sorts of systems is to bang the "Normalization of deviance is not okay" drum in every meeting. All failures need to go to the PM and get signed off on, otherwise the PM has a false sense of the actual reliability of the system. If the PM wants to get more budget for safety concerns, they should be able to hand a stack of 100s of pages of papers to whomever controls the purse strings and say "These are the failures in the last N days" If all they can say is "some of my engineers have expressed concerns" then 0 change will happen.
MaxfordAndSons 7y ago
Seems unlikely given that they were likely contractors: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...

cheez 7y ago

I took ethics courses as part of comp sci. Definitely left an imprint on me.

joe_the_user 7y ago

The reason the engineers are given ethical training is because engineers (like accountants, appraisers or similar people) have historically had an institutional independence from their employers. A civil engineer in particular can choose and will not to sign off on a given project that they know will be unsafe. Given that engineers are certified by their profession, their employment situation is not precarious and so they can reasonable make this refusal (this is degrading like all professions naturally but still). Programmers have no such institutional support. If a programmer refuses a job, it goes to someone else that's it. Programmers may have ethics but ethical training a la engineers isn't going to give them any leverage for choices.

luckydata 7y ago
Exactly! You seem to get it, the issue is not the lack of knowledge of ethics, but the lack of power. That's one of the many, many reasons why strong unions are so important and in this country we see the effects of not having them. Without protection for people refusing to do bad things, you create a system when there's always someone desperate, hungry or unethical enough to do things that shouldn't be done.
CobrastanJorji 7y ago
This sounds reasonable, but I don't quite understand that last bit. If a civil engineer refuses to sign off on a project, won't the same thing happen? Why won't the job go to the next civil engineer down the line?
- azernik 7y ago
  It's a collective action problem - the company can't fire the first engineer if they know everyone else they hire would also refuse orders. But each individual engineer has an incentive (not getting fired) to break with the group's strategy. This collective action problem is solved by coordination, through the means of the licensing body. That body can impose severe penalties (not just firing you from your current job, but from all future jobs) for anyone who betrays the group strategy, so an individual engineer can feel some more safety refusing orders in the knowledge that the whole profession will back them up. EDIT: In civil engineering, this system is propped up by the state, which requires plans to be signed off by a licensed engineer. The guild functions in this capacity as a subcontractor of the state, taking on a regulatory burden and allowing rather more severe punishments (barring someone from a profession) than would be acceptable from a purely state organ. In software, this could be enforced by similar means for safety-critical applications - the ACM, for example, could be required to license any software engineer, with the understanding that they would revoke licenses for negligence or malfeasance that didn't rise to the level or criminal liability.

jimbob45 7y ago

You don't need to take an ethics class to see that cutting corners when human lives are involved is a bad idea.

tbrownaw 7y ago
I think I heard somewhere that the entire purpose of Engineering as a whole, is to know which corners are safe to cut. "Anyone can build a bridge that stays up. Only an engineer can build a bridge that barely stays up." (I mean, it's obviously exaggerated for effect, but still.)

isthis1984 7y ago

Membership of the ACM has a code of ethics https://www.acm.org/code-of-ethics https://www.acm.org/code-of-ethics

aiisjustanif 7y ago

I don’t know if ASCE or NSPE are governing bodies as much they are societies. I think the Core of Engineers has way more power there from my experience in Civil. For Software that would be Homeland, FBI, and even the FAA for governing bodies specifically. But software doesn’t have ethically governing bodies because we have enough laws to cover a lot of mandatory auditing. For example: HIPAA - Medical Data Protection & Compliance Gramm-Leach-Bliley Act - Financial Data Protection & Compliance FISMA - Federal Data Compliance GDPR - PII Data Compliance PCI DSS - Credit Card Data Compliance Specific example of use: Code that is deployed that involves PCI data being handled requires a code review. There are heavy penalties for lost of these things, and I promise there are many companies focusing on this. But it could be very much improved with a proper governing body.