5 ms·
A place I once worked had their servers in-house. They had UPS on the roof, duel everything (fibre links etc). Apparently there wasn't a single point of failure
by aidos 9y ago
A place I once worked had their servers in-house. They had UPS on the roof, duel everything (fibre links etc). Apparently there wasn't a single point of failure in the system.
One day, the servers just vanish. We go into the server room to check. Someone had plugged all the machines into the same multi box / power extension cable and the fuse in it had popped.
Sometimes all your planning can be undone - especially once humans are involved :-)
- digi_owl 9y agoMakes me think of the story where the cleaning people would unplug a server rack to power their carpet cleaner.
- joejerryronnie 9y agoHaha, happened to me! Didn't see this until after I posted a reply to the parent :)
- digi_owl 9y agoHeh. The story i recall their initial problem was that the failure as always after office hours, but they could not find any fault with the hardware or software. So as a last desperate attempt, the person telling it basically sat watch inside the server room and noticed the door opening, a arm reaching in, unplugging a socket near the door, and plugging in a cable from the hallway. The very next day security was switching the lock on that door.
- jacquesm 9y ago> Apparently there wasn't a single point of failure in the system. Redundant power supplies on alternate phases is a pretty common element in designing any kind of server setup. So this wasn't a setup with a 'single point of failure', it was a setup with a very explicit single point of failure (several, in fact). Someone linked to this excellent slide deck yesterday: https://c3.nasa.gov/dashlink/static/media/other/ObservedFailures1.html https://c3.nasa.gov/dashlink/static/media/other/ObservedFail... Worth an hour of your time, not to make you paranoid but more to make it plain how hard this sort of planning really is.
- sogen 9y agoThanks, great stuff, like the story of the email that couldn't be sent farther than 500 miles.
- protomyth 9y agoI remember a data center that supposedly had dual fibre links to the outside. Found out later (after someone cut the fibre) that they actually linked back up just outside of town. I understand lawyers were needed.
- makomk 9y agoA certain important data center near here has dual fibre links, as mandated by their redundancy requirements. They're installed along both sides of the same major road.
- dmurray 9y agoThat's not so bad. The typical mode of failure for fibre lines is not flooding or tornadoes or landslides or anything else that would affect both sides of the road at the same time. It's someone digging up the road, which is unlikely to happen on both sides simultaneously.
- mentat 9y ago"unlikely"
- FooHentai 9y ago'Fully Diverse Paths' is something we take great pains to verify with our WAN provider, for exactly that reason. Because it is said that if you are heading into the woods and worried you might get lost, take a length of fiber with you. If you get lost, simply bury the fiber in the ground. In five minutes a man in hi-viz with a backhoe will come along and dig it up.
- jacquesm 9y agoIn a pinch, a piece of CAT5 will do.
- trhway 9y agoAdd automatic fire suppression system. Which one day turned on for whatever reason .