4 ms·

Don’t overlook the advice regarding setting up alerting. A long time ago I was managing several devices which ingested logs for a large enterprise. One day one

Don’t overlook the advice regarding setting up alerting. A long time ago I was managing several devices which ingested logs for a large enterprise. One day one of the devices suddenly couldn’t handled the load and sent an alert that it was dropping log messages. After a lot of digging and headscratching it turned out the battery backup on the device’s hardware RAID card had failed so (IIRC) it was operating in a mode without the onboard memory cache and couldn’t handle the write load anymore. We had to run to the data center and replace the entire 2RU device because of this. If we had alerting set up on the backup batteries at least we might have been able to react quicker.

phire 6y ago
I've had worse. Someone has set the machine up with a raid 10 of SSDs and no monitoring. At some point the RAID controller battery died, increasing the load on the SSDs. Who knows when that happened, no monitoring. Then I guess the SSDs failed one by one. I have no idea in which order, or how long the gaps were between the failures. Like I said, it would have been nice to have some kind of monitoring. Eventually the 3rd SSD died, and the machine died, finally calling attention to the problem. To top it all off, it had recently been discovered that the backups were incomplete, and everyone was still arguing over the best way to close the backup hole. So we forced to pull the SSDs and send them off to professionals for recovery. I believe we spent several thousand for express recovery. So yeah. Don't forget monitoring/alerting.
- louwrentius 6y ago
  Thanks for sharing, that was probably a terrible experience.
- funnybeam 6y ago
  As I find myself frequently repeating - redundancy without monitoring is not redundancy
6nf 6y ago
Monitoring is critical. Without monitoring it's just a time bomb. You need to set alerts and you also need to audit and test the alerting system every few months. Is so easy to forget that your alerting system depends on your email server or some network port or something that gets changed or removed without updating the alert system.

phire 6y ago

I've had worse. Someone has set the machine up with a raid 10 of SSDs and no monitoring. At some point the RAID controller battery died, increasing the load on the SSDs. Who knows when that happened, no monitoring. Then I guess the SSDs failed one by one. I have no idea in which order, or how long the gaps were between the failures. Like I said, it would have been nice to have some kind of monitoring. Eventually the 3rd SSD died, and the machine died, finally calling attention to the problem. To top it all off, it had recently been discovered that the backups were incomplete, and everyone was still arguing over the best way to close the backup hole. So we forced to pull the SSDs and send them off to professionals for recovery. I believe we spent several thousand for express recovery. So yeah. Don't forget monitoring/alerting.

louwrentius 6y ago
Thanks for sharing, that was probably a terrible experience.
funnybeam 6y ago
As I find myself frequently repeating - redundancy without monitoring is not redundancy

6nf 6y ago

Monitoring is critical. Without monitoring it's just a time bomb. You need to set alerts and you also need to audit and test the alerting system every few months. Is so easy to forget that your alerting system depends on your email server or some network port or something that gets changed or removed without updating the alert system.