4 ms·

I don't think we should really be looking at drive MTBF, but rather some measure of total expected loss. The bigger your drive, the more catastrophic the loss

I don't think we should really be looking at drive MTBF, but rather some measure of total expected loss. The bigger your drive, the more catastrophic the loss in the event of a failure, so even if the MTBF of a 2x capacity drive is only slightly higher, your total risk is more than 2x as high.

lallysingh 10y ago
Only if you don't have backups. If you do have backups, the cost of loss is proportional to your I/O rate for time spent in recovery.
- mdorazio 10y ago
  Hence the suggestion of RAID elsewhere in this thread, which means you might as well get more smaller drives in an array since the $/TB seems not to be dropping on these bigger drives.
  - epistasis 10y ago
    RAID is a really bad idea with larger disks, as 1)not all sources of failure are uncorrelated 2) with huge disks, the rebuild time to replace that failed disk gets really really long (I would guess 2-3 days to fully write to one of these disks. Edit: rebuilding a RAID rather than writing straight will take several times longer.) 3) during rebuild all the disks are taxed quite heavily, especially if they are still performing their typical day to day IO. The larger the disk, the more of them you want in a non-RAID unified erasure coded pool in order to manage these failures more gracefully.
    - lmm 10y ago
      I just rebuilt a raidz2 (4x4TB disks) on my home "server". Took a couple of days. I've never had even a single disk fail during rebuilding to date, so I'm not super worried about the risk of having 2x failures during the 4 days or so it takes to replace a disk (next-day delivery on the replacement and then doing the rebuild). It's a lot more "just works" than backups would be - in principle I'm not worried about high availability as such, but I am worried about faff when replacing.
      - epistasis 10y ago
        4TB with raid-z2 is perfectly acceptable, as it has lots of extra bits for the amount of down time. Also, it's ZFS, not straight RAID, which has advantages for bit flips. I still have many many 4TB drives in large RAID6 arrays. Personally, I'd stop at 4TB drives for this though. With 16TB disks, that rebuild time goes to 4x as long; and though one could probably write 16TB in 2-3 days in one go, a RAID rebuild of more than a week sounds typical. If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you.
        phil21 10y ago
        > If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you. The type of array redundancy (ZFS, mdadm, megaraid, etc.) does not matter. It is never a substitute for backups. End of discussion. These are two entirely different technologies meant to protect against entirely different problems. This is a common meme I've seen constantly through my entire career, and one of those common misconceptions you have to educate customers on quite often. RAID is not for data security. RAID is for availability. Snapshots, checksumming, etc. on ZFS are wonderful - I use it extensively. I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. Even if you could somehow guarantee ZFS or your hardware would never combine to lose your data (you can't) - it still doesn't protect from your house burning down. A substantial number of people completely neglect that failure mode until it's too late.
        epistasis 10y ago
        Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.
        lmm 10y ago
    - nradov 10y ago
      RAID 6 copes with this reasonably well. https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6
gcp 10y ago
I don't think this works. With more drives, you expect more drives to die in the same time period, so total expected loss stays exactly constant.
- mdorazio 10y ago
  Yes, but they don't all die at the same time, so you can utilize a RAID or similar array to mitigate the failures at a small cost in total storage space. Ex. my 5-drive RAID array runs 5x 3TB drives and provides about 12TB of usable space. If any single drive fails, it's a quick swap with a cheap replacement. If I instead had a single 12TB drive fail, I'd be screwed. Even if I didn't have a RAID array, any single drive failure would be a maximum of 3TB lost, vs. losing everything with a single drive. Basically it's much easier to minimize your data loss risk profile if you spread your data across multiple drives instead of relying on a single drive. Of course, if you need 100TB+ of storage, you might as well put these new 16TB drives in an array and save on the controller costs.
  - zzalpha 10y ago
    And then if there's an unrecoverable read error on one of those other massive drives during the rebuild, you're in deep trouble.. and the bigger the drive the more likely this is to happen (reaching near-certainty once the drive is large enough): http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/ http://www.zdnet.com/article/why-raid-5-stops-working-in-200...
    - snuxoll 10y ago
      Which is why the 3 and 4TB drives I have in my FreeNAS box at home are configured as mirrored pairs, the time it takes to resliver a drive in this configuration is already fairly long - RAID-Z or the hardware equivalent of RAID-5 would take upwards of days to complete and that's a disaster waiting to happen.
  - cmurf 10y ago
    At this size, the rebuild time is a day, or even 2-3 days if it's actively being used during the rebuild. It still makes sense to do at least a 2x raid1, which rebuilds faster than a cluster over wired ethernet. But if you have this much online storage, you probably have a high availability need also, which implies cluster in case a storage brick loses a power supply or logic board. It's not just a drive that can fail.

lallysingh 10y ago

Only if you don't have backups. If you do have backups, the cost of loss is proportional to your I/O rate for time spent in recovery.

mdorazio 10y ago
Hence the suggestion of RAID elsewhere in this thread, which means you might as well get more smaller drives in an array since the $/TB seems not to be dropping on these bigger drives.
- epistasis 10y ago
  RAID is a really bad idea with larger disks, as 1)not all sources of failure are uncorrelated 2) with huge disks, the rebuild time to replace that failed disk gets really really long (I would guess 2-3 days to fully write to one of these disks. Edit: rebuilding a RAID rather than writing straight will take several times longer.) 3) during rebuild all the disks are taxed quite heavily, especially if they are still performing their typical day to day IO. The larger the disk, the more of them you want in a non-RAID unified erasure coded pool in order to manage these failures more gracefully.
  - lmm 10y ago
    I just rebuilt a raidz2 (4x4TB disks) on my home "server". Took a couple of days. I've never had even a single disk fail during rebuilding to date, so I'm not super worried about the risk of having 2x failures during the 4 days or so it takes to replace a disk (next-day delivery on the replacement and then doing the rebuild). It's a lot more "just works" than backups would be - in principle I'm not worried about high availability as such, but I am worried about faff when replacing.
    - epistasis 10y ago
      4TB with raid-z2 is perfectly acceptable, as it has lots of extra bits for the amount of down time. Also, it's ZFS, not straight RAID, which has advantages for bit flips. I still have many many 4TB drives in large RAID6 arrays. Personally, I'd stop at 4TB drives for this though. With 16TB disks, that rebuild time goes to 4x as long; and though one could probably write 16TB in 2-3 days in one go, a RAID rebuild of more than a week sounds typical. If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you.
      - phil21 10y ago
        > If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you. The type of array redundancy (ZFS, mdadm, megaraid, etc.) does not matter. It is never a substitute for backups. End of discussion. These are two entirely different technologies meant to protect against entirely different problems. This is a common meme I've seen constantly through my entire career, and one of those common misconceptions you have to educate customers on quite often. RAID is not for data security. RAID is for availability. Snapshots, checksumming, etc. on ZFS are wonderful - I use it extensively. I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. Even if you could somehow guarantee ZFS or your hardware would never combine to lose your data (you can't) - it still doesn't protect from your house burning down. A substantial number of people completely neglect that failure mode until it's too late.
        epistasis 10y ago
        Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.
        lmm 10y ago
  - nradov 10y ago
    RAID 6 copes with this reasonably well. https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6

gcp 10y ago

I don't think this works. With more drives, you expect more drives to die in the same time period, so total expected loss stays exactly constant.

mdorazio 10y ago
Yes, but they don't all die at the same time, so you can utilize a RAID or similar array to mitigate the failures at a small cost in total storage space. Ex. my 5-drive RAID array runs 5x 3TB drives and provides about 12TB of usable space. If any single drive fails, it's a quick swap with a cheap replacement. If I instead had a single 12TB drive fail, I'd be screwed. Even if I didn't have a RAID array, any single drive failure would be a maximum of 3TB lost, vs. losing everything with a single drive. Basically it's much easier to minimize your data loss risk profile if you spread your data across multiple drives instead of relying on a single drive. Of course, if you need 100TB+ of storage, you might as well put these new 16TB drives in an array and save on the controller costs.
- zzalpha 10y ago
  And then if there's an unrecoverable read error on one of those other massive drives during the rebuild, you're in deep trouble.. and the bigger the drive the more likely this is to happen (reaching near-certainty once the drive is large enough): http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/ http://www.zdnet.com/article/why-raid-5-stops-working-in-200...
  - snuxoll 10y ago
    Which is why the 3 and 4TB drives I have in my FreeNAS box at home are configured as mirrored pairs, the time it takes to resliver a drive in this configuration is already fairly long - RAID-Z or the hardware equivalent of RAID-5 would take upwards of days to complete and that's a disaster waiting to happen.
- cmurf 10y ago
  At this size, the rebuild time is a day, or even 2-3 days if it's actively being used during the rebuild. It still makes sense to do at least a 2x raid1, which rebuilds faster than a cluster over wired ethernet. But if you have this much online storage, you probably have a high availability need also, which implies cluster in case a storage brick loses a power supply or logic board. It's not just a drive that can fail.