6 ms·
>On the evening of Saturday, January 26th, our database server had three hard drives fail. It was designed to handle two disk failures, but three failed disks m
by akerro 8y ago
>On the evening of Saturday, January 26th, our database server had three hard drives fail. It was designed to handle two disk failures, but three failed disks made the situation catastrophic.
Like in the joke.
Our data storage was so well designed, that when first disk died, we didn't notice. When second disk died, we also didn't notice. When third disk died, we noticed.
- howiroll 8y agoIt’s likely that the three drives died concurrently.
- FPGAhacker 8y agoAnd they are probably setting themselves up for a similar fate in the future by replacing them all (failed and working alike), and at once.
- lucb1e 8y agoThey don't fail that reliably, unfortunately. It's not a light bulb with a certain number of hours on it. See Backblaze's drive statistics. Edit: I just saw it's about ssds and not hdds, but while Backblaze might not have stats on those, I'm fairly sure my comment still applies. Not 100%, but I assume they account for failures due to predicable issues like write wear.
- nemothekid 8y ago>I'm fairly sure my comment still applies There are other comments in this thread that go into this, but your comment doesn't apply. Another poster used the light bulb analogy well to describe SSD failures. It's very common to see SSDs that were purchased at the same time, that were likely manufactured in the same batch, fail within hours of each other. Im' pretty sure I even read this fact on Backblaze.
- raverbashing 8y agoAt least they should try and get from different vendors/models
- eknkc 8y agoHow? I mean, I can see a power issue taking out all at once. It could be that. What other way would 3 disks fail at once?
- lucb1e 8y agoLuck? And hence it makes the news because such bad luck messes things up? We all know drives fail, I'm not surprised this happened somewhere on the planet. I would be surprised if it happened to me.
- icelancer 8y agoAll SSDs were from the same batch and had the same amount of writes on them and all failed in a very small window. It's a known issue with consumer drives.
- majewsky 8y agoGood to see that I'm not paranoid for explicitly using drives of different vendors in my RAID at home.
- mirimir 8y agoIs there a good Linux tool for logging RAID status? I just have scripts that write key data from "mdadm -D /dev/md$N" to logs. But then I need to remember to review the logs regularly.