3 ms·

This line of thinking is just an extension of everyday thinking of far too many people. Take nuclear reactors as an example. Some people seem to legitimately t

This line of thinking is just an extension of everyday thinking of far too many people. Take nuclear reactors as an example. Some people seem to legitimately think that its possible to eliminate all sources of errors in such an immensely complex system. If there's been an accident, the analysis is generally "we have to add sections in the manual for this specific scenario and we'll be safe", rather than accepting that it's impossible to prevent all errors and that it's merely a question of time before the next anomaly arises.

gunfighthacksaw 5y ago
The trick is to make ‘time before the next anomaly’ large enough to allow future generations a fair crack at solving/preventing it.
inglor_cz 5y ago
"Some people seem to legitimately think that its possible to eliminate all sources of errors in such an immensely complex system." That mostly describes amateurs. Designers of modern reactors generally think along the "let us minimize possible negative effects of a failure" lines. So, no positive void coefficient (a horrible feature of RBMK in Chernobyl), addition of core catchers [0] etc. [0] https://en.wikipedia.org/wiki/Core_catcher https://en.wikipedia.org/wiki/Core_catcher
- beaconstudios 5y ago
  yes this, good design not only includes means of avoiding failure, but means of failing safely.
Jtsummers 5y ago
People working in systems safety work on (at least) two levels: 1. Physical 2. Social On the physical side they try to establish controls that prevent or mitigate issues or minimize the consequences of them. For instance, a modern reactor (probably, not my domain) is going to be designed in a way that certain meltdown situations can't occur (physically designed to separate reactor materials at certain thresholds, resulting in an automatic shutdown versus manual) or are properly contained (smaller reactors where a runaway reaction is more easily contained with modern materials at a reasonable cost). On the social side there is training, regulatory controls, and an emphasis on safety culture/discipline. Discipline doesn't scale, but you can't totally avoid having it. This would be things like not turning off an alarm because it disrupts your nap (looking at you, Homer J. Simpson), but actually responding to it to determine its legitimacy (related concept: normalization of deviance). They'd also want to look at it from an HCI perspective and minimize false alarms and overwhelming operators with alarms or information that seem to be at the same level of importance, but aren't. So true critical issues are brought to the front as high priority, but less critical issues (ones that should be addressed but are lower priority) won't overwhelm the operator. (I've worked with others on this wrt avionics systems, alarm fatigue is real so you have to be choosy in which things get audible alarms and which get visual alarms.) > If there's been an accident, the analysis is generally "we have to add sections in the manual for this specific scenario and we'll be safe", In my experience, that's a belief held by the prideful and the novices. > rather than accepting that it's impossible to prevent all errors and that it's merely a question of time before the next anomaly arises. Anyone with a modicum of humility or experience will accept this and act accordingly.
jeltz 5y ago
I do not get your example. The same applied to bridges, cars, airplanes, trains, space crafts, hydro electric plants, skyscrapers, ... When building complex things anomalies will happen which means we should follow good engineering practices and add extra safety systems.
pdimitar 5y ago
Not the best possible analogy but I get your point. The problem with OpenSSL and Heartbleed was there weren't even procedures in case of sh_t hitting the fan, whereas with nuclear reactors that's often the case. In programming, and especially systems programming, people love to pretend they are infallible, to the detriment of everyone suffering from permanent firmware bugs and critical security infrastructure leaking their secrets.

gunfighthacksaw 5y ago

The trick is to make ‘time before the next anomaly’ large enough to allow future generations a fair crack at solving/preventing it.

inglor_cz 5y ago

"Some people seem to legitimately think that its possible to eliminate all sources of errors in such an immensely complex system." That mostly describes amateurs. Designers of modern reactors generally think along the "let us minimize possible negative effects of a failure" lines. So, no positive void coefficient (a horrible feature of RBMK in Chernobyl), addition of core catchers [0] etc. [0] https://en.wikipedia.org/wiki/Core_catcher https://en.wikipedia.org/wiki/Core_catcher

beaconstudios 5y ago
yes this, good design not only includes means of avoiding failure, but means of failing safely.

Jtsummers 5y ago

People working in systems safety work on (at least) two levels: 1. Physical 2. Social On the physical side they try to establish controls that prevent or mitigate issues or minimize the consequences of them. For instance, a modern reactor (probably, not my domain) is going to be designed in a way that certain meltdown situations can't occur (physically designed to separate reactor materials at certain thresholds, resulting in an automatic shutdown versus manual) or are properly contained (smaller reactors where a runaway reaction is more easily contained with modern materials at a reasonable cost). On the social side there is training, regulatory controls, and an emphasis on safety culture/discipline. Discipline doesn't scale, but you can't totally avoid having it. This would be things like not turning off an alarm because it disrupts your nap (looking at you, Homer J. Simpson), but actually responding to it to determine its legitimacy (related concept: normalization of deviance). They'd also want to look at it from an HCI perspective and minimize false alarms and overwhelming operators with alarms or information that seem to be at the same level of importance, but aren't. So true critical issues are brought to the front as high priority, but less critical issues (ones that should be addressed but are lower priority) won't overwhelm the operator. (I've worked with others on this wrt avionics systems, alarm fatigue is real so you have to be choosy in which things get audible alarms and which get visual alarms.) > If there's been an accident, the analysis is generally "we have to add sections in the manual for this specific scenario and we'll be safe", In my experience, that's a belief held by the prideful and the novices. > rather than accepting that it's impossible to prevent all errors and that it's merely a question of time before the next anomaly arises. Anyone with a modicum of humility or experience will accept this and act accordingly.

jeltz 5y ago

I do not get your example. The same applied to bridges, cars, airplanes, trains, space crafts, hydro electric plants, skyscrapers, ... When building complex things anomalies will happen which means we should follow good engineering practices and add extra safety systems.

pdimitar 5y ago

Not the best possible analogy but I get your point. The problem with OpenSSL and Heartbleed was there weren't even procedures in case of sh_t hitting the fan, whereas with nuclear reactors that's often the case. In programming, and especially systems programming, people love to pretend they are infallible, to the detriment of everyone suffering from permanent firmware bugs and critical security infrastructure leaking their secrets.