4 ms·

Not everything, just things that can kill hundreds to thousands of people if they go wrong. That is safety engineering 101; identify the important and critical

by Veserv 3y ago

Not everything, just things that can kill hundreds to thousands of people if they go wrong. That is safety engineering 101; identify the important and critical parts and do those parts better and more carefully. You do not need to uniformly improve all elements to the same high standards as long as you can isolate them from the safety-critical path. This is a necessity because, as you point out, doing everything to the high standards is economically infeasible. But doing everything to economical standards is mass murder. No one standard works for the entire system, so you must isolate and apply the correct amount of effort for each component. This is a very weird concept for most software developers because most software is a big mushy ball of spaghetti where it is impossible to isolate or even identify the critical path. Luckily, those individuals largely do not develop safety-critical systems, so the lack of knowledge of basic safety processes does not hinder their deployments or result in a danger to public safety. What is problematic is when those people start doing safety critical systems. That is how you get the ideas being satirized by the article. Acting as if there is nothing in between total recklessness and impossibly perfect safety, so there is no choice but criminal recklessness. Basic industry-standard processes are, in fact, like 6 orders of magnitude safer during development and increase development velocity toward a safe, working product. Any viewpoint as insane as normalizing deaths during the development process is totally incompatible with successfully deploying a safety critical product. Deaths may be unavoidable due to the sheer weight of statistics, but deaths are always a failure of immense gravity. A flippant disregard for human life has no place on a safety critical product.

JumpinJack_Cash 3y ago
> > Not everything, just things that can kill hundreds to thousands of people if they go wrong The original Al Quaeda plan for the WTC was to detonate explosives to tear it down, they tried in the 90s but of course they were so inept that they just made a huge mess in the parking lot. However let's say they succeded. Do you think we'd have checkpoints looking for explosives (or even components to make explosives) while entering Manhattan and all the 1000 odd major global cities which exceed 1M population? There is a thing with flight. Planes look so big and so gross and like they should not be able to fly and so people are scared of them, fear of flight is one of the most common fear. In the post 9/11 world add to that being under a plane, in the proximity of a plane or in a builing from which a plane can be seen. None of this stuff happens with trains and they too can kill thousands.
- Veserv 3y ago
  I am not really sure what your point is. I think the argument thread is: 1. Root claims aviation processes are a good example of safety processes that self driving cars and other safety-critical industries can adopt. 2. You disagree claiming that aviation processes are too onerous citing TSA and ground security processes. In addition, you state “If everything were to be as safe as aviation the whole world would come to a screeching halt standstill.” You are implying that the only choice is between aviation processes everywhere or nowhere. 3. I pointed out that high safety processes only need to be applied where it matters. You can apply challenging processes on a case by case basis depending on the risk. 4. I guess you are now stating that aviation processes are excessive over reactions because in no other industry would they do the same things? And thus they are a poor model that should not be adopted by other industries since aviation is too risk averse? To which I say, okay? Aviation is not the only model of safe processes, it is just a model with a very good safety record, probably the best. It is useful for establishing what an upper bound on safety processes might look like and cost. But, as you point out, the cost/benefit may not make sense for other industries. The point is that there is a vast range of processes between nothing and aviation. Something closer to aviation or trains rather than total recklessness is probably more fit for safety critical systems that can kill hundreds to thousands of people.
  - JumpinJack_Cash 3y ago
    > > The point is that there is a vast range of processes between nothing and aviation. Something closer to aviation or trains rather than total recklessness is probably more fit for safety critical systems that can kill hundreds to thousands of people. To me the point is that there are many things that are used by billions of people and can kill thousands of people per incident when things go wrong, if even one such system has zero safety (for whatever reason ranging from economics to culture) then it makes sense to roll down safety of every other such system to zero as well to pursue quality of life between now and the not so distant moment when something goes wrong
    - Veserv 3y ago
      So if you safety processes made 99% of situations 1,000,000 times safer, but 1% of situations remain high risk you may as well throw out all your processes and incur 1,000,000 times more risk 100% of the time? That is positively absurd. It might make sense to reevaluate what is the most efficient allocation of resources which would result in the greatest holistic safety increase per resource spent, but throwing everything out to reach the lowest common denominator is just straight up nonsense. The only way that makes sense is if you assume a completely uniform risk tolerance, completely uniform mitigation ROI, and you assume the highest risk activity is the true underlying uniform risk tolerance with the other 99% of low risk activities being the outliers. Literally every one of those assumptions is ludicrous.

by Veserv 3y ago

JumpinJack_Cash 3y ago

> > Not everything, just things that can kill hundreds to thousands of people if they go wrong The original Al Quaeda plan for the WTC was to detonate explosives to tear it down, they tried in the 90s but of course they were so inept that they just made a huge mess in the parking lot. However let's say they succeded. Do you think we'd have checkpoints looking for explosives (or even components to make explosives) while entering Manhattan and all the 1000 odd major global cities which exceed 1M population? There is a thing with flight. Planes look so big and so gross and like they should not be able to fly and so people are scared of them, fear of flight is one of the most common fear. In the post 9/11 world add to that being under a plane, in the proximity of a plane or in a builing from which a plane can be seen. None of this stuff happens with trains and they too can kill thousands.

Veserv 3y ago
I am not really sure what your point is. I think the argument thread is: 1. Root claims aviation processes are a good example of safety processes that self driving cars and other safety-critical industries can adopt. 2. You disagree claiming that aviation processes are too onerous citing TSA and ground security processes. In addition, you state “If everything were to be as safe as aviation the whole world would come to a screeching halt standstill.” You are implying that the only choice is between aviation processes everywhere or nowhere. 3. I pointed out that high safety processes only need to be applied where it matters. You can apply challenging processes on a case by case basis depending on the risk. 4. I guess you are now stating that aviation processes are excessive over reactions because in no other industry would they do the same things? And thus they are a poor model that should not be adopted by other industries since aviation is too risk averse? To which I say, okay? Aviation is not the only model of safe processes, it is just a model with a very good safety record, probably the best. It is useful for establishing what an upper bound on safety processes might look like and cost. But, as you point out, the cost/benefit may not make sense for other industries. The point is that there is a vast range of processes between nothing and aviation. Something closer to aviation or trains rather than total recklessness is probably more fit for safety critical systems that can kill hundreds to thousands of people.
- JumpinJack_Cash 3y ago
  > > The point is that there is a vast range of processes between nothing and aviation. Something closer to aviation or trains rather than total recklessness is probably more fit for safety critical systems that can kill hundreds to thousands of people. To me the point is that there are many things that are used by billions of people and can kill thousands of people per incident when things go wrong, if even one such system has zero safety (for whatever reason ranging from economics to culture) then it makes sense to roll down safety of every other such system to zero as well to pursue quality of life between now and the not so distant moment when something goes wrong
  - Veserv 3y ago
    So if you safety processes made 99% of situations 1,000,000 times safer, but 1% of situations remain high risk you may as well throw out all your processes and incur 1,000,000 times more risk 100% of the time? That is positively absurd. It might make sense to reevaluate what is the most efficient allocation of resources which would result in the greatest holistic safety increase per resource spent, but throwing everything out to reach the lowest common denominator is just straight up nonsense. The only way that makes sense is if you assume a completely uniform risk tolerance, completely uniform mitigation ROI, and you assume the highest risk activity is the true underlying uniform risk tolerance with the other 99% of low risk activities being the outliers. Literally every one of those assumptions is ludicrous.