4 ms·

Nuclear criticality safety was one of my favorite courses back during my nuclear engineering degree. The failure modes, like the article mentions, can be incred

Nuclear criticality safety was one of my favorite courses back during my nuclear engineering degree. The failure modes, like the article mentions, can be incredibly non-intuitive. Swirling vortices, firefighting foam, and even the mere presence of a human body can lead to a fatal power excursion. A lot of complex computational modeling and simulation goes into determining safety limits for fissile systems, as well as a lot of sheer imagination. The most important thing I took away from that experience is the concept of engineered compliance. People will break administrative rules and regulations all the time if they get int the way of work, so you need to engineer the physical system to account for that (e.g. flat or skinny geometry, or lots of neutron absorbers). This can apply to any compliance activity related to safety and security, so the lessons are universal. Los Alamos published a good survey on various worldwide criticality accidents (PDF): https://www.nrc.gov/docs/ML0037/ML003731912.pdf https://www.nrc.gov/docs/ML0037/ML003731912.pdf. Worth a read for all the various ways things can go wrong.

jiggawatts 3y ago
> People will break administrative rules and regulations all the time if they get in the way of work The example I like to use is that the more paperwork you require for something like "obtain an internal use certificate for API endpoints" then the direct consequence will be less encryption of internal API endpoints. Note that I didn't say "there may be", or "I predict that". No. It's a guaranteed consequence, like gradient descent making AIs better at meeting loss function. Except that it's gradient ascent, and you're making it steeper and expecting people to climb harder instead of just going around your pointless obstacle. "More paperwork" -> "Less security" "Less paperwork" -> "More security" Choose.
- formerly_proven 3y ago
  Another typical example is charging internal applications for IDPs, guess what, people are just gonna use LDAP/AD directly, at a not-so-small detriment to application and user security. These are things where cost center accounting just plainly hurts the company. (There are many cases where it works well and aligns incentives).
- AnimalMuppet 3y ago
  Same thing with problem reports and maintenance requests. The more work you make them, the more people will put up with things that kind of halfway work. That's bad for the productivity of the organization as a whole, though, no matter whether the more cumbersome process makes things more convenient for IT or management or auditors.
- hilbert42 3y ago
  "More paperwork" -> "Less security" "Less paperwork" -> "More security" That's almost an axiom in some circumstances but with criticality accidents there's a difference—first, the stakes don't get much higher, at close quarters death's a certainty and it's about as unpleasant as it gets; second, it's all over and irreversible in milliseconds. There is no time to think, the body-brain reaction time is far too slow to take corrective action no matter how effective it may be when deployed. That's different to giving consideration to a course of action based on the difficulty of obstacles in one's way—choosing the path of least resistance, etc. The issues with that infamous photo are different, there was no obvious logical if decision to be made—it was all very straightforward, lay the Pu out and photograph it. The problems, however, are multifold: (a) those involved didn't have sufficient understanding of the possible consequences (of the physics involved), (b) they were poorly trained in safety procedures, and (c), the work environment wasn't 'engineered' to ensure compliance—that is, nothing stopped them from doing it. As I pointed out in my earlier post, training is crucial and it cannot be just theoretical. When the stakes are this high training has to be ingrained to such a degree that certain procedures are automatic, similarly, dangerous situations don't just become obvious after thinking about them, they are so the instant the brain responds. For instance, one recognizes the danger as quickly as one would if one tripped over a snake. That's what the military does with training, one automatically reacts to a situation with the correct response without thinking. And that takes practice until one's perfect. It works, it's a damn long time since I did any military training and I can stil carry out certain actions and procedures I learned decades ago and do so automatically and without a moment's thought. The people who set up and took those photos should have been trained to a similar level of discipline.
- pixl97 3y ago
  That's why you monitor and block every non encrypted API as quickly as possible.
  - jiggawatts 3y ago
    That’s a career-limiting move in most places. If actually enforced this way, then the next consequence is that people will use a credit card to spin their service up in some random cloud that you don’t control and won’t break. … where all API endpoints are encrypted by default with zero paperwork required. Get it? Forcing people to climb steeper slopes never ever works. You need to make sure they fall into the pit of success even if they’re stumbling around in the dark with their eyes closed.
killjoywashere 3y ago
This is a good place to add to the reading list: strongly recommend Command and Control as well, by Eric Schlosser: https://www.amazon.com/Command-Control-Damascus-Accident-Illusion/dp/0143125788 https://www.amazon.com/Command-Control-Damascus-Accident-Ill...

jiggawatts 3y ago

> People will break administrative rules and regulations all the time if they get in the way of work The example I like to use is that the more paperwork you require for something like "obtain an internal use certificate for API endpoints" then the direct consequence will be less encryption of internal API endpoints. Note that I didn't say "there may be", or "I predict that". No. It's a guaranteed consequence, like gradient descent making AIs better at meeting loss function. Except that it's gradient ascent, and you're making it steeper and expecting people to climb harder instead of just going around your pointless obstacle. "More paperwork" -> "Less security" "Less paperwork" -> "More security" Choose.

formerly_proven 3y ago
Another typical example is charging internal applications for IDPs, guess what, people are just gonna use LDAP/AD directly, at a not-so-small detriment to application and user security. These are things where cost center accounting just plainly hurts the company. (There are many cases where it works well and aligns incentives).
AnimalMuppet 3y ago
Same thing with problem reports and maintenance requests. The more work you make them, the more people will put up with things that kind of halfway work. That's bad for the productivity of the organization as a whole, though, no matter whether the more cumbersome process makes things more convenient for IT or management or auditors.
hilbert42 3y ago
"More paperwork" -> "Less security" "Less paperwork" -> "More security" That's almost an axiom in some circumstances but with criticality accidents there's a difference—first, the stakes don't get much higher, at close quarters death's a certainty and it's about as unpleasant as it gets; second, it's all over and irreversible in milliseconds. There is no time to think, the body-brain reaction time is far too slow to take corrective action no matter how effective it may be when deployed. That's different to giving consideration to a course of action based on the difficulty of obstacles in one's way—choosing the path of least resistance, etc. The issues with that infamous photo are different, there was no obvious logical if decision to be made—it was all very straightforward, lay the Pu out and photograph it. The problems, however, are multifold: (a) those involved didn't have sufficient understanding of the possible consequences (of the physics involved), (b) they were poorly trained in safety procedures, and (c), the work environment wasn't 'engineered' to ensure compliance—that is, nothing stopped them from doing it. As I pointed out in my earlier post, training is crucial and it cannot be just theoretical. When the stakes are this high training has to be ingrained to such a degree that certain procedures are automatic, similarly, dangerous situations don't just become obvious after thinking about them, they are so the instant the brain responds. For instance, one recognizes the danger as quickly as one would if one tripped over a snake. That's what the military does with training, one automatically reacts to a situation with the correct response without thinking. And that takes practice until one's perfect. It works, it's a damn long time since I did any military training and I can stil carry out certain actions and procedures I learned decades ago and do so automatically and without a moment's thought. The people who set up and took those photos should have been trained to a similar level of discipline.
pixl97 3y ago
That's why you monitor and block every non encrypted API as quickly as possible.
- jiggawatts 3y ago
  That’s a career-limiting move in most places. If actually enforced this way, then the next consequence is that people will use a credit card to spin their service up in some random cloud that you don’t control and won’t break. … where all API endpoints are encrypted by default with zero paperwork required. Get it? Forcing people to climb steeper slopes never ever works. You need to make sure they fall into the pit of success even if they’re stumbling around in the dark with their eyes closed.

killjoywashere 3y ago

This is a good place to add to the reading list: strongly recommend Command and Control as well, by Eric Schlosser: https://www.amazon.com/Command-Control-Damascus-Accident-Illusion/dp/0143125788 https://www.amazon.com/Command-Control-Damascus-Accident-Ill...