5 ms·
That culture of pressure never changed, and they are better for it. The best analogy I can give you is this: Imagine your CTO just decides to walk up to your d
by brandmeyer 7y ago
That culture of pressure never changed, and they are better for it.
The best analogy I can give you is this: Imagine your CTO just decides to walk up to your datacenter to throw open a breaker or few to verify that your failover and redundancy tech works correctly.
The CTO doesn't do this often - just once every few months. Each time the failure mode is different. Remove a live hard drive, disconnect some network switches, etc. The only reason she does it to is to emphasize the importance of these drills upon her VPs and directors.
They run drills of this nature on their teams every few days.
- sien 7y agoI actually had a boss who was a CTO who would do something like this. He was a good boss. At another job I did something daft and tried to recover a backup. The backups hadn't actually been recoverable for at least 2 years.
- solotronics 7y agoThis happens to us regularly without someone having to go play chaos monkey.
- kstenerud 7y agoSimulating a power failure to test and prove the resiliency of the system is precisely what caused the Chernobyl disaster.
- killjoywashere 7y agoAh, no. The cause of Chernobyl is lost to the dead but probably caused by a young engineer who didn't quite do the right thing. You absolutely must drill for rare, high stakes events. Auto-rotations in helicopters. Mass casualties in hospitals. Reactor scrams. Man overboard on a ship. All these things are drilled because you don't have time to read the book when it actually happens, and it rarely happens.
- kstenerud 7y agohttps://en.wikipedia.org/wiki/Chernobyl_disaster https://en.wikipedia.org/wiki/Chernobyl_disaster The accident started during a safety test on an RBMK-type nuclear reactor, which was commonly used throughout the Soviet Union. The test was a simulation of an electrical power outage to aid the development of a safety procedure for maintaining reactor cooling water circulation until the back-up electrical generators could provide power. This gap was about one minute and had been identified as a potential safety problem that could cause the nuclear reactor core to overheat. It was hoped to prove that the residual rotational energy in a turbine generator could provide enough power to cover the gap. Three such tests had been conducted since 1982, but they had failed to provide a solution. On this fourth attempt, an unexpected 10-hour delay meant that an unprepared operating shift was on duty. During the planned decrease of reactor power in preparation for the electrical test, the power unexpectedly dropped to a near-zero level. The operators were able to only partially restore the specified test power, which put the reactor in a potentially unstable condition. This risk was not made evident in the operating instructions, so the operators proceeded with the electrical test. Upon test completion, the operators triggered a reactor shutdown, but a combination of unstable conditions and reactor design flaws caused an uncontrolled nuclear chain reaction instead.