4 ms·

We were writing it to 48,000 hard drives (we did not use the full capacity of these disks, though), and every time we ran our sort, at least one of our disks ma

by nebula 18y ago

We were writing it to 48,000 hard drives (we did not use the full capacity of these disks, though), and every time we ran our sort, at least one of our disks managed to break (this is not surprising at all given the duration of the test, the number of disks involved, and the expected lifetime of hard disks). I'm a bit at lost here. Does that mean that one in ~50K hard disks fail within six hours of usage?

russell 18y ago
Not at all surprising given the large number of disks involved. Here is a very interesting analysis of actual Google failure rates: http://storagemojo.com/2007/02/19/googles-disk-failure-experience/ http://storagemojo.com/2007/02/19/googles-disk-failure-exper...
- nebula 18y ago
  Thanks for the pointer; it's interesting.
wmf 18y ago
1 failure per 50K in 6 hours is the same as 1 per 34 in a year, or a 3% annual failure rate. I think this is the same number Google reported in their disk failure paper.
- nebula 18y ago
  Actually, I did this math before I posted my comment. What I don't understand is this: With x% annual failure rate, are the failures uniformly distributed over time? I would expect that it's probably a normal distribution over time. (i.e., no failures to very few failures in the beginning, and the failures keep increasing with time).
  - bd 18y ago
    It's probable that they didn't use 48,000 brand new hard drives, but reused whatever they already had in their datacenters. Which would have a mixture of different drive ages, I guess.
    - nebula 18y ago
      That's an interesting possibility. But the fact that the article specifically mentions about drive failures and that they use some of RAID to cope with this makes me wonder if they were using old disks. It's probably easier to get all new drives and use them isn't it? On second thoughts though, it might be much easier to tell GFS to make an extra mirror than to make sure that all disks are brand new. So, i buy your line of thinking :)

by nebula 18y ago

russell 18y ago

Not at all surprising given the large number of disks involved. Here is a very interesting analysis of actual Google failure rates: http://storagemojo.com/2007/02/19/googles-disk-failure-experience/ http://storagemojo.com/2007/02/19/googles-disk-failure-exper...

nebula 18y ago
Thanks for the pointer; it's interesting.

wmf 18y ago

1 failure per 50K in 6 hours is the same as 1 per 34 in a year, or a 3% annual failure rate. I think this is the same number Google reported in their disk failure paper.

nebula 18y ago
Actually, I did this math before I posted my comment. What I don't understand is this: With x% annual failure rate, are the failures uniformly distributed over time? I would expect that it's probably a normal distribution over time. (i.e., no failures to very few failures in the beginning, and the failures keep increasing with time).
- bd 18y ago
  It's probable that they didn't use 48,000 brand new hard drives, but reused whatever they already had in their datacenters. Which would have a mixture of different drive ages, I guess.
  - nebula 18y ago
    That's an interesting possibility. But the fact that the article specifically mentions about drive failures and that they use some of RAID to cope with this makes me wonder if they were using old disks. It's probably easier to get all new drives and use them isn't it? On second thoughts though, it might be much easier to tell GFS to make an extra mirror than to make sure that all disks are brand new. So, i buy your line of thinking :)