4 ms·

Reminder: Have backups. Imagine losing 32TB of data in one go.

by fb03 3y ago

Dalewyn 3y ago
Reminder: RAID is not a backup. You're going to have more drives die while you're resilvering 32TB of data.
- XorNot 3y ago
  But "RAID is not a backup" is not related to this problem though. "RAID is not a backup" s a reflection of the fact that your RAID array is in one location, and instantly mirrors all data copied to it. A disaster to your storage array outside of a disk failure, will annihilate all your data (and this used to include "RAID controller dies"). The practical issues of RAID rebuilds also don't quite work that way - the "drives die while rebuilding problem" has two components: (1) is that beyond a certain size, the risk of bit-error while resilvering becomes a certainty due to all the data you're reading - ZFS fixes this problem with it's checksums basically. (2) If all your disks were installed at the same time, then the chance of another disk failure is high because they're likely all in the same part of the mean-time-to-failure distribution. (3) the risk of an error is higher because you're doing a lot more activity then normal, and so the additional stress is increasing the chance of triggering a failure.
- HumanOstrich 3y ago
  Do you not regularly scrub your arrays to surface these issues promptly?
  - mbreese 3y ago
    With 32TB, nothing would be done “promptly”. Can you imagine how long resilvering this drive would take? These aren’t feasible for RAID setups…
    - Dylan16807 3y ago
      If you go from double to triple parity it should far more than make up for the extra resilvering time. Though if you want specific numbers, what would you estimate as the per-drive odds of 8TB and 40TB drives dying during a resilver? Let's say they take 2 and 7 days respectively.
      - sp332 3y ago
        Maybe, but then these have to be a lot cheaper for a triple redundant setup to be cost effective.
        Dylan16807 3y ago
        Not really. Let's say you were doing 8+2 before (times five) with 8TB drives, and switch to 8+3 with 40TB drives. You go from 400TB of raw space to 440TB of raw space, but you also go from 50 drive slots down to 11. Even if both drive models are the same cost per terabyte, the latter setup should have a lower total cost. But I would also expect these drives to be cheaper per TB, at least by the time HAMR is 2-3 generations old.
        sp332 3y ago
        I think if you're in the market for 32TB drives, you're already using 20TB ones instead of 8. With 8+2 at 20TB you pay for 200TB and get 160TB usable. With 7+3 at 32TB you pay for 320TB and have 224TB available. That's 160% of TB paid for and only 140% of TB usable. HMR will need to be a sizable 12.5% discount per terabyte just to break even there. If you only do 5+3 to keep the same 160T available, it's 256TB or 128% paid for and the same 100% usable, so an even steeper 22% discount to break even.
        Dylan16807 3y ago
        If you're going from 20TB to 32TB then it barely affects your RAID reliability at all. You don't need to increase parity in that case. And nobody that's price conscious would make a 5+3 RAID so I'm not very worried about such an extreme scenario.
mmwelt 3y ago
Restoring backups on a 32TB drive would also take a very long time! Other comments have mentioned that it would probably take more than 13 hours to fill the drive. On a related note, remirroring or rebuilding a 32TB RAID drive would probably take days.
- toomuchtodo 3y ago
  Backup to cloud provider, and then have them ship you a replacement drive with the data preloaded when your local drive goes bad (Backblaze, for example, does this). Local drives are more fast cache for data that should be stored reliably elsewhere (maybe hot, maybe cold, depending on provider and cost concerns).
alwayslikethis 3y ago
It also takes forever to restore them. The issue with ever increasing HDD speeds is they don't get any faster. It takes literal days to get all the data off them, and if you try to resilver your array when a drive fails, it's more likely another will fail too. Some of these big drives have fairly low endurance ratings, so they might fail earlier than you think [1] 1. https://www.servethehome.com/discussing-low-wd-red-pro-nas-hard-drive-endurance-ratings/ https://www.servethehome.com/discussing-low-wd-red-pro-nas-h...

Reminder: Have backups. Imagine losing 32TB of data in one go.

by fb03 3y ago

Reminder: Have backups. Imagine losing 32TB of data in one go.

Dalewyn 3y ago

Reminder: RAID is not a backup. You're going to have more drives die while you're resilvering 32TB of data.

XorNot 3y ago
But "RAID is not a backup" is not related to this problem though. "RAID is not a backup" s a reflection of the fact that your RAID array is in one location, and instantly mirrors all data copied to it. A disaster to your storage array outside of a disk failure, will annihilate all your data (and this used to include "RAID controller dies"). The practical issues of RAID rebuilds also don't quite work that way - the "drives die while rebuilding problem" has two components: (1) is that beyond a certain size, the risk of bit-error while resilvering becomes a certainty due to all the data you're reading - ZFS fixes this problem with it's checksums basically. (2) If all your disks were installed at the same time, then the chance of another disk failure is high because they're likely all in the same part of the mean-time-to-failure distribution. (3) the risk of an error is higher because you're doing a lot more activity then normal, and so the additional stress is increasing the chance of triggering a failure.
HumanOstrich 3y ago
Do you not regularly scrub your arrays to surface these issues promptly?
- mbreese 3y ago
  With 32TB, nothing would be done “promptly”. Can you imagine how long resilvering this drive would take? These aren’t feasible for RAID setups…
  - Dylan16807 3y ago
    If you go from double to triple parity it should far more than make up for the extra resilvering time. Though if you want specific numbers, what would you estimate as the per-drive odds of 8TB and 40TB drives dying during a resilver? Let's say they take 2 and 7 days respectively.
    - sp332 3y ago
      Maybe, but then these have to be a lot cheaper for a triple redundant setup to be cost effective.
      - Dylan16807 3y ago
        Not really. Let's say you were doing 8+2 before (times five) with 8TB drives, and switch to 8+3 with 40TB drives. You go from 400TB of raw space to 440TB of raw space, but you also go from 50 drive slots down to 11. Even if both drive models are the same cost per terabyte, the latter setup should have a lower total cost. But I would also expect these drives to be cheaper per TB, at least by the time HAMR is 2-3 generations old.
        sp332 3y ago
        I think if you're in the market for 32TB drives, you're already using 20TB ones instead of 8. With 8+2 at 20TB you pay for 200TB and get 160TB usable. With 7+3 at 32TB you pay for 320TB and have 224TB available. That's 160% of TB paid for and only 140% of TB usable. HMR will need to be a sizable 12.5% discount per terabyte just to break even there. If you only do 5+3 to keep the same 160T available, it's 256TB or 128% paid for and the same 100% usable, so an even steeper 22% discount to break even.
        Dylan16807 3y ago
        If you're going from 20TB to 32TB then it barely affects your RAID reliability at all. You don't need to increase parity in that case. And nobody that's price conscious would make a 5+3 RAID so I'm not very worried about such an extreme scenario.

mmwelt 3y ago

Restoring backups on a 32TB drive would also take a very long time! Other comments have mentioned that it would probably take more than 13 hours to fill the drive. On a related note, remirroring or rebuilding a 32TB RAID drive would probably take days.

toomuchtodo 3y ago
Backup to cloud provider, and then have them ship you a replacement drive with the data preloaded when your local drive goes bad (Backblaze, for example, does this). Local drives are more fast cache for data that should be stored reliably elsewhere (maybe hot, maybe cold, depending on provider and cost concerns).

alwayslikethis 3y ago

It also takes forever to restore them. The issue with ever increasing HDD speeds is they don't get any faster. It takes literal days to get all the data off them, and if you try to resilver your array when a drive fails, it's more likely another will fail too. Some of these big drives have fairly low endurance ratings, so they might fail earlier than you think [1] 1. https://www.servethehome.com/discussing-low-wd-red-pro-nas-hard-drive-endurance-ratings/ https://www.servethehome.com/discussing-low-wd-red-pro-nas-h...