3 ms·
I remember explaining to a storage customer why they needed three very expensive switches at the core of their network. One to handle the traffic, another to ha
by ChuckMcM 10y ago
I remember explaining to a storage customer why they needed three very expensive switches at the core of their network. One to handle the traffic, another to handle the traffic when the first was busy, and a third which was in the rack, patched and updated to one release behind the main switches, that could be swapped in by moving cables in under 5 minutes of operator time.
They initially thought I was kidding but then we did the fail tree on a white board. It was an interesting experience for them, understanding what it took to get what they took for granted.
- jerf 10y agoYou used the term "fail tree" which sounds interesting and I tried to learn more on the Internet but hit problems. Do you mean fault tree? https://en.wikipedia.org/wiki/Fault_tree_analysis https://en.wikipedia.org/wiki/Fault_tree_analysis If not, do you have a better search term and/or link I could follow up on?
- ChuckMcM 10y agoFault trees seem to be reasonably close. While developing my thinking and understanding on reliability analysis I came at it from an analog of decision trees. There were a couple of influential talks that got me started, one was by Sandia labs discussing the ways in which they insure that nuclear devices can't be detonated without approval (big requirement) through a process called vulnerability analysis, and another on how Citibank worked their network to insure uptime. The Chaos Monkey series from Netflix was quite fun as well. Google did something similar internally with its DiRT exercises (Diaster Recovery). My "fail tree" (and I'll put it in quotes as unique to my conception of them) analysis consists of identifying a failure, the system response to the failure, a time to fix for the failed system, and a guess at the uncertainty on the fix. So for example "switch hardware failure" is a failure, with a fix time that varies based on "replacement part on hand" to "order/ship/install (replace) the entire switch". The first order is failure/fix tree with callouts of down time. The second order is mitigation/cost with mitigation strategies and their cost resulting in a new call out of potential downtime, and the third order is mitigation accelerators and their cost (which shorten recovery to non-degraded mode) which affect cost and possible down time. Much of that you can do on paper, but sometimes you will have to run experiments to see how long things take to fix.
- jerf 10y agoThank you.