3 ms·
In safety-critical systems, we distinguish between accidents (actual loss, e.g. lives, equipment, etc.) and hazardous states. The equation is hazardous state +
by kqr 1y ago
In safety-critical systems, we distinguish between accidents (actual loss, e.g. lives, equipment, etc.) and hazardous states. The equation is
hazardous state + environmental conditions = accident
Since we can only control the system, and not its environment, we focus on preventing hazardous states, rather than accidents. If we can keep the system out of all hazardous states, we also avoid accidents. (Trying to prevent accidents while not paying attention to hazardous states amounts to relying on the environment always being on our side, and is bound to fail eventually.)
One such hazardous state we have defined in aviation is "less than N minutes of fuel remaining when landing". If an aircraft lands with less than N minutes of fuel on board, it would only have taken bad environmental conditions to make it crash, rather than land. Thus we design commercial aviation so that planes always have N minutes of fuel remaining when landing. If they don't, that's a big deal: they've entered a hazardous state, and we never want to see that. (I don't remember if N is 30 or 45 or 60 but somewhere in that region.)
For another example, one of my children loves playing around cliffs and rocks. Initially he was very keen on promising me that he wouldn't fall down. I explained the difference between accidents and hazardous states to him in childrens' terms, and he realised slowly that he cannot control whether or not he has an accident, so it's a bad idea to promise me that he won't have an accident. What he can control is whether or not bad environmental conditions lead to an accident, and he does that by keeping out of hazardous states. In this case, the hazardous state would be standing less than a child-height within a ledge when there is nobody below ready to catch. He can promise me to avoid that, and that satisfies me a lot more than a promise to not fall.
- jama211 1y agoWell said, will think about asking this attitude towards my child, seems very helpful
- jacquesm 1y agoIf you haven't done so: please write a book. Aim it towards software professionals in non-regulated industries. I promise to buy 50 to give to all of my software developing colleagues. As for 'N', for turboprops it is 45, for jets it is 30.
- edanm 1y agoSeconded! This was an extremely well written and well thought out explanation of this idea. Would love to read more along these lines. (Will now be checking out your blog.)
- jacquesm 1y agoAlso check out risks digest: https://catless.ncl.ac.uk/Risks/ https://catless.ncl.ac.uk/Risks/
- xeonmc 1y agoIf he aims it toward five year olds as he had explained it, bet it would be even more applicable to our profession.
- abustamam 1y agoHaving spent some time with my five year old nieces and nephews, sometimes I wonder if five year olds could run companies better. (note: obviously sarcastic but kids really do have some amazing insights that we forget when trying to chase KPIs or revenue)
- xeonmc 1y agoSee also: various points in the Evil Overlord list[0]. Selected examples: #12: One of my advisors will be an average five-year-old child. Any flaws in my plan that he is able to spot will be corrected before implementation. #60: My five-year-old child advisor will also be asked to decipher any code I am thinking of using. If he breaks the code in under 30 seconds, it will not be used. Note: this also applies to passwords. #74: When I create a multimedia presentation of my plan designed so that my five-year-old advisor can easily understand the details, I will not label the disk "Project Overlord" and leave it lying on top of my desk. [0] https://tvtropes.org/pmwiki/pmwiki.php/Main/EvilOverlordList https://tvtropes.org/pmwiki/pmwiki.php/Main/EvilOverlordList
- 1y ago
- AnthonyMouse 1y ago> Trying to prevent accidents while not paying attention to hazardous states amounts to relying on the environment always being on our side, and is bound to fail eventually. The reason they had less than 30 minutes of fuel was because the environment wasn't on their side. They started out with a normal amount of reserve and then things went quite badly and the reserve was sufficient but only just. The question then is, how much of an outlier was this? Was this a perfect storm that only happens once in a century and the thing worse than this that would actually have exhausted the reserve only happens once in ten centuries? Or are planes doing this every Tuesday which would imply that something is very wrong?
- kqr 1y agoThis is why staying out of hazardous conditions is a dynamic control problem, rather than a simple equation or plan you can set up ahead of time. There are multiple controllers interacting with the system (the FADEC computer in the engines, the flight management computer in the plane, pilots, ground crew, dispatchers, air traffic controllers, the people at EASA drafting regulations, etc.), trying to keep it outside of hazardous conditions. They do so by observing the state the system and the environment is in ("feedback"), running simulations of how it will evolve in the future ("mental models"), and making adjustments to the system ("control inputs") to keep it outside of hazardous conditions. Whenever the system enters a hazardous condition, there was something that made these controllers insufficient. Either someone had inadequate feedback, or inadequate mental models, or the control inputs were inoperational or insufficient. Or sometimes an entire controller that ought to have been there was missing! In this case it seems like the hazard could have been avoided any number of ways: ground the plane, add more fuel, divert sooner, be more conservative about weather on alternates, etc. Which control input is appropriate and how to ensure it is enacted in the future is up to the real investigators with access to all data necessary. ----- You are correct that we will not ever be able to set up a system where all controllers are able to always keep it out of hazardous states perfectly. If that was a thing we would never have any accident ever – we would only have intentional losses that are calculated to be worth their revenue in additional efficiency. But by adopting the right framework for thinking about this ("how do active controllers dynamically keep the system out of hazards?") we can do a pretty good job of preventing most such problems. The good news is that predicting hazardous states is much easier than predicting accidents, so we can actually do a lot of this design up-front without first having an accident happen and then learning from it.
- ndsipa_pomu 1y agoThat's very enlightening. I'm casually interested in traffic safety and road/junction designs from the perspective of a UK cyclist and there's a lot to be learnt from the safety culture/practices of the aviation industry. I typically think in terms of "safety margins" whilst cycling (e.g. if a driver pulls out of a side road in front of me, how quickly can I avoid them via swerving or brake to avoid a collision). I can imagine that hazardous states can be applied to a lot of the traffic behaviour at junctions.