4 ms·

Hence the suggestion of RAID elsewhere in this thread, which means you might as well get more smaller drives in an array since the $/TB seems not to be dropping

Hence the suggestion of RAID elsewhere in this thread, which means you might as well get more smaller drives in an array since the $/TB seems not to be dropping on these bigger drives.

epistasis 10y ago
RAID is a really bad idea with larger disks, as 1)not all sources of failure are uncorrelated 2) with huge disks, the rebuild time to replace that failed disk gets really really long (I would guess 2-3 days to fully write to one of these disks. Edit: rebuilding a RAID rather than writing straight will take several times longer.) 3) during rebuild all the disks are taxed quite heavily, especially if they are still performing their typical day to day IO. The larger the disk, the more of them you want in a non-RAID unified erasure coded pool in order to manage these failures more gracefully.
- lmm 10y ago
  I just rebuilt a raidz2 (4x4TB disks) on my home "server". Took a couple of days. I've never had even a single disk fail during rebuilding to date, so I'm not super worried about the risk of having 2x failures during the 4 days or so it takes to replace a disk (next-day delivery on the replacement and then doing the rebuild). It's a lot more "just works" than backups would be - in principle I'm not worried about high availability as such, but I am worried about faff when replacing.
  - epistasis 10y ago
    4TB with raid-z2 is perfectly acceptable, as it has lots of extra bits for the amount of down time. Also, it's ZFS, not straight RAID, which has advantages for bit flips. I still have many many 4TB drives in large RAID6 arrays. Personally, I'd stop at 4TB drives for this though. With 16TB disks, that rebuild time goes to 4x as long; and though one could probably write 16TB in 2-3 days in one go, a RAID rebuild of more than a week sounds typical. If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you.
    - phil21 10y ago
      > If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you. The type of array redundancy (ZFS, mdadm, megaraid, etc.) does not matter. It is never a substitute for backups. End of discussion. These are two entirely different technologies meant to protect against entirely different problems. This is a common meme I've seen constantly through my entire career, and one of those common misconceptions you have to educate customers on quite often. RAID is not for data security. RAID is for availability. Snapshots, checksumming, etc. on ZFS are wonderful - I use it extensively. I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. Even if you could somehow guarantee ZFS or your hardware would never combine to lose your data (you can't) - it still doesn't protect from your house burning down. A substantial number of people completely neglect that failure mode until it's too late.
      - epistasis 10y ago
        Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.
      - lmm 10y ago
        It's a question of costs and benefits. RAID absolutely does reduce the risk of data loss. The extent to which you want to take further steps is a question of how valuable your data is and how much of it there is. It can be perfectly legitimate to decide that, given a particular risk profile and a finite budget, you can't afford both RAID and backups and the former is better value than the latter.
        throwawayish 10y ago
        If I guess right in what you mean here, then I'd say that in these instances backups are better value than RAID.
  - barrkel 10y ago
    Make sure you're snapshotting with exponential-style backoff (e.g. keep 24 hours of hourly snapshots, 7 days of daily snapshots, a few weeks of weekly snapshots, etc.) to guard against clobbering or deleting data. ZFS isn't an absolute panacea. It can get very confused if you have a systematic problem one day - e.g. if you ever have intermittent bus problems (e.g. dodgy backplane), it can start to believe that that blocks are permanently damaged and it's hard to convince it to read through to the disk storage. I had to copy the superblock from an old snapshot[1] once because it was convinced all of its copies were bad. This wasn't during writes; it was during a zfs scrub. If you don't do it regularly, act like a zfs scrub could potentially wipe out all your data. This is more likely on home hardware, without ECC RAM and less likely to be built to handle loads of concurrent I/O throughput. Keep on using backups. [1] Thankfully, because of my snapshotting policy, my old snapshot wasn't actually very old.
- nradov 10y ago
  RAID 6 copes with this reasonably well. https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6

Hence the suggestion of RAID elsewhere in this thread, which means you might as well get more smaller drives in an array since the $/TB seems not to be dropping on these bigger drives.

epistasis 10y ago

RAID is a really bad idea with larger disks, as 1)not all sources of failure are uncorrelated 2) with huge disks, the rebuild time to replace that failed disk gets really really long (I would guess 2-3 days to fully write to one of these disks. Edit: rebuilding a RAID rather than writing straight will take several times longer.) 3) during rebuild all the disks are taxed quite heavily, especially if they are still performing their typical day to day IO. The larger the disk, the more of them you want in a non-RAID unified erasure coded pool in order to manage these failures more gracefully.

lmm 10y ago
I just rebuilt a raidz2 (4x4TB disks) on my home "server". Took a couple of days. I've never had even a single disk fail during rebuilding to date, so I'm not super worried about the risk of having 2x failures during the 4 days or so it takes to replace a disk (next-day delivery on the replacement and then doing the rebuild). It's a lot more "just works" than backups would be - in principle I'm not worried about high availability as such, but I am worried about faff when replacing.
- epistasis 10y ago
  4TB with raid-z2 is perfectly acceptable, as it has lots of extra bits for the amount of down time. Also, it's ZFS, not straight RAID, which has advantages for bit flips. I still have many many 4TB drives in large RAID6 arrays. Personally, I'd stop at 4TB drives for this though. With 16TB disks, that rebuild time goes to 4x as long; and though one could probably write 16TB in 2-3 days in one go, a RAID rebuild of more than a week sounds typical. If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you.
  - phil21 10y ago
    > If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you. The type of array redundancy (ZFS, mdadm, megaraid, etc.) does not matter. It is never a substitute for backups. End of discussion. These are two entirely different technologies meant to protect against entirely different problems. This is a common meme I've seen constantly through my entire career, and one of those common misconceptions you have to educate customers on quite often. RAID is not for data security. RAID is for availability. Snapshots, checksumming, etc. on ZFS are wonderful - I use it extensively. I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. Even if you could somehow guarantee ZFS or your hardware would never combine to lose your data (you can't) - it still doesn't protect from your house burning down. A substantial number of people completely neglect that failure mode until it's too late.
    - epistasis 10y ago
      Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.
    - lmm 10y ago
      It's a question of costs and benefits. RAID absolutely does reduce the risk of data loss. The extent to which you want to take further steps is a question of how valuable your data is and how much of it there is. It can be perfectly legitimate to decide that, given a particular risk profile and a finite budget, you can't afford both RAID and backups and the former is better value than the latter.
      - throwawayish 10y ago
        If I guess right in what you mean here, then I'd say that in these instances backups are better value than RAID.
- barrkel 10y ago
  Make sure you're snapshotting with exponential-style backoff (e.g. keep 24 hours of hourly snapshots, 7 days of daily snapshots, a few weeks of weekly snapshots, etc.) to guard against clobbering or deleting data. ZFS isn't an absolute panacea. It can get very confused if you have a systematic problem one day - e.g. if you ever have intermittent bus problems (e.g. dodgy backplane), it can start to believe that that blocks are permanently damaged and it's hard to convince it to read through to the disk storage. I had to copy the superblock from an old snapshot[1] once because it was convinced all of its copies were bad. This wasn't during writes; it was during a zfs scrub. If you don't do it regularly, act like a zfs scrub could potentially wipe out all your data. This is more likely on home hardware, without ECC RAM and less likely to be built to handle loads of concurrent I/O throughput. Keep on using backups. [1] Thankfully, because of my snapshotting policy, my old snapshot wasn't actually very old.
nradov 10y ago
RAID 6 copes with this reasonably well. https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6