I've had worse. Someone has set the machine up with a raid 10 of SSDs and no monitoring. At some point the RAID controller battery died, increasing the load o

I've had worse. Someone has set the machine up with a raid 10 of SSDs and no monitoring. At some point the RAID controller battery died, increasing the load on the SSDs. Who knows when that happened, no monitoring. Then I guess the SSDs failed one by one. I have no idea in which order, or how long the gaps were between the failures. Like I said, it would have been nice to have some kind of monitoring. Eventually the 3rd SSD died, and the machine died, finally calling attention to the problem. To top it all off, it had recently been discovered that the backups were incomplete, and everyone was still arguing over the best way to close the backup hole. So we forced to pull the SSDs and send them off to professionals for recovery. I believe we spent several thousand for express recovery. So yeah. Don't forget monitoring/alerting.