I used to have a couple of filers with thousands of disks on premise, we'd always have a dead disk somewhere. Keeping up with dead disks was almost a full time

I used to have a couple of filers with thousands of disks on premise, we'd always have a dead disk somewhere. Keeping up with dead disks was almost a full time job, you'd open a case with the vendor, get them access to the datacenter, then be on the phone to guide them to the rack and verify that the new disk came online. To not go completely crazy, we usually batched these jobs once or twice per week. And even with double parity and replicated filers in two locations things can go wrong. Also a favorite of mine is power cycling a machine. With enough disks, chances are high that a couple won't come back after they spin down.