4 ms·
Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people runni
by seanp2k2 10y ago
Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through.
One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.
The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.
- hackuser 10y ago> The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
- protomyth 10y agoHad a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
- throwawayish 10y agoEspecially true for backups. Make sure those aren't write-only!
- shubb 10y agoNote to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
- protomyth 10y agoGood luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots. It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
- syntheticnature 10y ago"The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe." --https://en.wikipedia.org/wiki/Systemantics#System_failure https://en.wikipedia.org/wiki/Systemantics#System_failure
- bcbrown 10y agoI love that book. Have you read https://www.amazon.com/Introduction-General-Systems-Thinking-Anniversary/dp/0932633498 https://www.amazon.com/Introduction-General-Systems-Thinking...? It's similar, but a bit more rigorous.
- throwawayish 10y ago> The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed. This is the area were inverted Unix philosophy applies: Give me good news as well. A weekly mail, that backups (or the filer or RAID or whatever) works A-OK. If the mail stops coming, something broke.