5 ms·
Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are al
by dekhn 7y ago
Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before.
Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past the cage.
- gerdesj 7y agoI walk the server room daily, every morning. I've tended our monitoring system for 15 years now and I don't trust myself to be infallible. I'm also the MD ...
- deleted 7y ago[deleted]
- jiveturkey 7y agoMD = VP for those not in finance
- bennyelv 7y agoIn the UK MD = CEO (when not finance), so without more context you couldn’t really say. He’s either an MD in finance, which is the equivalent of a VP (roughly) in other kinds of companies, or he’s the CEO.
- killjoywashere 7y agoAdmiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.
- MaxBarraclough 7y agoI'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?
- killjoywashere 7y agoIf you are responsible for building the industry that designs and builds nuclear submarines that carry nuclear missiles, you had better make sure that those submarines and their crews can handle chaos monkeys. Also, Rickover was Congress's favorite admiral. They forced the Navy to promote him. I'm pretty sure they made sure that the laws were to his liking.
- GoodJokes 7y agoLol, as software developers it should be obvious that you can test stuff like this without endangering, uh, the world. This general sounds like a pompous tool. Like most millitary higher ups.
- nexuist 7y agoIn general, one would hope that a nuclear system is designed such that the problem can easily be corrected if a single button or lever is accidentally pressed. It would be quite a terrible system if you could e.g. trigger a meltdown with just one action.
- bovermyer 7y agoA reactor scram is basically just an emergency shutdown. If I were a nuclear engineer, it might give me a heart attack to hear the scram alarms, but I would be plenty happy knowing the scram works.
- dekhn 7y agoRickover was the sort of person who had the technical expertise, gumption, and charisma to get away with this. It's worth reading up on this amazing individual, who made navy's nuclear reactors so safe, and led the creation of nuclear reactor expertise within the navy. Also, if you're not confident enough in your nuclear reactor to apply chaos monkey techniques, you shouldn't be engineering nuclear reactors.