4 ms·

Yes. Some systems can be made more accountable. They can provide traceability from their inputs to output, provide reflexive access to source code they run, a

by michael1999 2y ago

Yes. Some systems can be made more accountable. They can provide traceability from their inputs to output, provide reflexive access to source code they run, and provide evidentiary traces for reliability. Safety critical systems that operate faster than human reactions are not accountable. So that's why we never make them responsible. So who is? Same as for bridges that fall down -- the engineers. People forget that civil engineers sign up for accountability that could lead to serious civil or even criminal liability. Which is exactly the point of this aphorism. Boeing was facing criminal charges, and is currently under a consent decree for exactly this kind of sloppy systems work.

taeric 2y ago
I can agree that there is a lot of lifting for "management decision" on the page, but my point is pretty strictly that it is overstated. I'm largely used to this getting discussed much more generically. That is, just as I am ok with AES on cars, I am largely ok with the idea that systems can, in fact, be designed in such a way that they could rise to the level of accountability that we would want them to have. I'm ok with the idea that, at the time of that manual, it was not obvious that systems would grow to have more durable storage than makes sense. But, I'd expect that vehicles and anything with automated systems should have a system log that is available and can be called up for evidence. And yes, Boeing was facing criminal charges. As they should. I don't think it should be a witch hunt for individuals at Boeing for being the last or first on the line to sign something.
- michael1999 2y ago
  Don't ignore the importance of the phrase "management decision". When the AES fails to activate because of a bug in the firmware, can you interrogate the machine? No. Can you ask the oem why they hired an intern to write the code and no test plan? No. Can you ask the brake why its design was chosen over others? No. That kind of accountability is not possible in those kinds of systems. And that's why proper engineering is a profession with standards, traceability, and accountability. If an executive at Boeing directed supervisory staff to falsify logs to maintain throughput, and a traceable failure led to fatalities, I expect that individual to be criminally charged.

taeric 2y ago

I can agree that there is a lot of lifting for "management decision" on the page, but my point is pretty strictly that it is overstated. I'm largely used to this getting discussed much more generically. That is, just as I am ok with AES on cars, I am largely ok with the idea that systems can, in fact, be designed in such a way that they could rise to the level of accountability that we would want them to have. I'm ok with the idea that, at the time of that manual, it was not obvious that systems would grow to have more durable storage than makes sense. But, I'd expect that vehicles and anything with automated systems should have a system log that is available and can be called up for evidence. And yes, Boeing was facing criminal charges. As they should. I don't think it should be a witch hunt for individuals at Boeing for being the last or first on the line to sign something.

michael1999 2y ago
Don't ignore the importance of the phrase "management decision". When the AES fails to activate because of a bug in the firmware, can you interrogate the machine? No. Can you ask the oem why they hired an intern to write the code and no test plan? No. Can you ask the brake why its design was chosen over others? No. That kind of accountability is not possible in those kinds of systems. And that's why proper engineering is a profession with standards, traceability, and accountability. If an executive at Boeing directed supervisory staff to falsify logs to maintain throughput, and a traceable failure led to fatalities, I expect that individual to be criminally charged.