9 ms·

So a quick calculation shows that it will take roughly two full weeks for one complete disk write, like what would happen when rebuilding an array. I guess an

So a quick calculation shows that it will take roughly two full weeks for one complete disk write, like what would happen when rebuilding an array. I guess an even higher level of redundancy will be required at these capacities.

rini17 6y ago
Or a filesystem like zfs/btrfs that can heal itself quicker by spreading writes between more drives in exchange for (temporary) free space imbalance. Not sure if they do actually implement such algorithm already.
- tecleandor 6y ago
  OpenZFS DRAID, that I think it's coming with v2.1, might help with this. https://news.ycombinator.com/item?id=26179566 https://news.ycombinator.com/item?id=26179566
88 6y ago
I wouldn’t feel comfortable with RAID5/6 even on today’s ~10TB drives. The solution isn’t more parity disks, it’s to move away from that RAID set up entirely.
- gbrown_ 6y ago
  > I wouldn’t feel comfortable with RAID5/6 even on today’s ~10TB drives. I fully agree with this part. De-clustered RAID really should have been mainstream a long time ago. Thankfully dRAID in OpenZFS will bring this to the "masses" in the sense of it being an open source implementation, whereas this has traditionally been a proprietary feature. > The solution isn’t more parity disks, it’s to move away from that RAID set up entirely. But I strongly disagree with RAID being the problem. Rather the increasing of capacity with bandwidth not really improving is the issue. Of course this is a inherent limitation of disk drives.
  - Robotbeat 6y ago
    They are adding multiple actuators, which actually should help that fundamental limitation.
    - altcognito 6y ago
      Could easily reduce reliability.
      - Robotbeat 6y ago
        It’s possible, just as increasing the density of NAND with 4 Levels per cell reduces write count. But can be countered by more careful engineering of the device.
        altcognito 6y ago
        NAND failures tend to be more localized though.
        Robotbeat 6y ago
        This is probably not representative because it was using low grade USB flash devices from 10-15 years ago, but I think I’ve had more NAND flash failures than hard drive failures. I don’t think doubling the number of heads would make a huge difference as you wouldn’t be quite halving the reliability.
        altcognito 6y ago
        Oh, I totally agree in that regard. Hard drive technology has been mature for a good number of decades, so a lot of the kinks have been worked out.
      - jacquesm 6y ago
        No, it will reduce reliability. More parts = worse reliability.
- nsgi 6y ago
  Depends what you use them for. With a high-capacity hard drive you could store a lot of downloaded movies and you can always download them again if it fails
  - 88 6y ago
    I think for most people having to download tens of terabytes of any content would be extremely challenging.
Cthulhu_ 6y ago
Yeah, these would be more for data warehousing; keep in mind that at the same time, SSD's are getting much bigger (while being more compact), so SSD's are taking over from hard drives on the 'front lines', while these new big disks can be used for big / slow storage.
api 6y ago
I'd skip RAID altogether with these and use a RAIN approach with multiple lightweight nodes with one drive each.
- effie 6y ago
  Why? It is cheaper to run redundant disks and power supplies with single mainboard than to run two mainboards.
  - api 6y ago
    The mainboard could be a Raspberry Pi class system but with an on-board SATA. There are a number of those on the market. The board would be smaller than the drive. Seagate should sell such a machine capable of running arbitrary Linux distributions as a kind of mini-NAS.
    - effie 6y ago
      Oh you mean a home solution for few people who do not require 99.99% uptime, then yeah one could try to go that way although all such attempts I've seen (such as [1]) seem way too cumbersome, I would just build a silent PC, way simpler, more extensible and easily replaceable components. I thought you meant to use RAIN for business servers. [1] https://www.jeffgeerling.com/blog/2020/building-fastest-raspberry-pi-nas-sata-raid https://www.jeffgeerling.com/blog/2020/building-fastest-rasp...
ChuckMcM 6y ago
The more interesting data point is that the "URE" spec. That is "unrecoverable read error". All of these drives do track at a time with full ECC to facilitate reading in the presence of noise, but a track write can have effects on adjacent tracks and temperature can affect head flying height, and vibration can affect head tracking, etc. All of which, statistically, can create a non-recoverable bit error. Typically this spec is like 1 in 10^14 bits. What it interesting about this spec is that it is statistical, meaning that the more times you read the bits, the more likely you are to hit a URE. HDD manufacturers do all sorts of things to avoid them (re-trys, track mirroring, etc) but ultimately it is a mechanical system and there are a LOT of bits. When I was at NetApp, the company would get reports of UREs that were fixed by RAID reconstruction (so you read a disk, you get an error and you fix it by the ECC in the other drives). And we had determined that by 4TB it would be "stupid" to using mirroring for protection since if a drive failed you couldn't count on being able to re-silver the mirror by getting a clean read of the other half. (something like 1 in 20 attempts would fail and you'd have data loss). And of course reconstructing a RAID-5 group for a failed drive you need to read the other drives, for larger groups of disks (which people did to maximize data storage) you started becoming at risk of being able to reconstruct all of the stripes. That was the genesis of doing dual parity (which Netapp called 'diagonal parity' because of the design). The ZFS folks have also considered this and designed both dual and TRIPLE parity which makes sense for these larger drives! It already takes a long time to reconstruct any "wide" vDev in ZFS that is zraid2. So one wonders if you could solve for device capacity where 'reconstruction takes 3 years' (the depreciated life span of a drive at Google).
- NateLawson 6y ago
  Kirk McKusick took this kind of scale into consideration when designing softupdates for UFS. One of his criticisms of the competing log structured filesystems at the time was that the replay process would require RAM in excess of any server's capacity once disk drives were large enough. He had designed the softupdates recovery process to be incremental and use a fixed amount of RAM that didn't scale with volume size. This kind of holistic systems' awareness is a sign of software that's designed well.
- effie 6y ago
Dylan16807 6y ago
You did the wrong calculation. Either you used a very pessimistic estimate for drive speed and forgot to factor in the increase in linear density, or you used a relatively accurate calculation and mixed up megabytes per second with megabits per second.
- effie 6y ago
  With 200MB/s I'm getting one week as the lower limit for rebuild time. If the drives are used heavily while rebuilding, this can go much longer.
  - Dylan16807 6y ago
    The increase in platter density means you should expect numbers more like 600MB/s. > If the drives are used heavily while rebuilding, this can go much longer. True but that's a separate factor and depends on how the array is laid out.
    - effie 6y ago
      You are right, let's hope those 600MB/s will be the case.
Mandatum 6y ago
It took 24 hours for me to validate/check my 16TB disk (I don't know the linux commands, it was a tool /r/DataHoarders suggested). I've upgraded the quality of all my video files and I've still got TB's of space to go.. 120TB.. Man, to those in the future laughing at this comment with your 120TB games and your 60TB super-high-definition VR feature length movies.. I'm jealous.

rini17 6y ago

Or a filesystem like zfs/btrfs that can heal itself quicker by spreading writes between more drives in exchange for (temporary) free space imbalance. Not sure if they do actually implement such algorithm already.

tecleandor 6y ago
OpenZFS DRAID, that I think it's coming with v2.1, might help with this. https://news.ycombinator.com/item?id=26179566 https://news.ycombinator.com/item?id=26179566

88 6y ago

I wouldn’t feel comfortable with RAID5/6 even on today’s ~10TB drives. The solution isn’t more parity disks, it’s to move away from that RAID set up entirely.

gbrown_ 6y ago
> I wouldn’t feel comfortable with RAID5/6 even on today’s ~10TB drives. I fully agree with this part. De-clustered RAID really should have been mainstream a long time ago. Thankfully dRAID in OpenZFS will bring this to the "masses" in the sense of it being an open source implementation, whereas this has traditionally been a proprietary feature. > The solution isn’t more parity disks, it’s to move away from that RAID set up entirely. But I strongly disagree with RAID being the problem. Rather the increasing of capacity with bandwidth not really improving is the issue. Of course this is a inherent limitation of disk drives.
- Robotbeat 6y ago
  They are adding multiple actuators, which actually should help that fundamental limitation.
  - altcognito 6y ago
    Could easily reduce reliability.
    - Robotbeat 6y ago
      It’s possible, just as increasing the density of NAND with 4 Levels per cell reduces write count. But can be countered by more careful engineering of the device.
      - altcognito 6y ago
        NAND failures tend to be more localized though.
        Robotbeat 6y ago
        This is probably not representative because it was using low grade USB flash devices from 10-15 years ago, but I think I’ve had more NAND flash failures than hard drive failures. I don’t think doubling the number of heads would make a huge difference as you wouldn’t be quite halving the reliability.
        altcognito 6y ago
        Oh, I totally agree in that regard. Hard drive technology has been mature for a good number of decades, so a lot of the kinks have been worked out.
    - jacquesm 6y ago
      No, it will reduce reliability. More parts = worse reliability.
nsgi 6y ago
Depends what you use them for. With a high-capacity hard drive you could store a lot of downloaded movies and you can always download them again if it fails
- 88 6y ago
  I think for most people having to download tens of terabytes of any content would be extremely challenging.

Cthulhu_ 6y ago

Yeah, these would be more for data warehousing; keep in mind that at the same time, SSD's are getting much bigger (while being more compact), so SSD's are taking over from hard drives on the 'front lines', while these new big disks can be used for big / slow storage.

api 6y ago

I'd skip RAID altogether with these and use a RAIN approach with multiple lightweight nodes with one drive each.

effie 6y ago
Why? It is cheaper to run redundant disks and power supplies with single mainboard than to run two mainboards.
- api 6y ago
  The mainboard could be a Raspberry Pi class system but with an on-board SATA. There are a number of those on the market. The board would be smaller than the drive. Seagate should sell such a machine capable of running arbitrary Linux distributions as a kind of mini-NAS.
  - effie 6y ago
    Oh you mean a home solution for few people who do not require 99.99% uptime, then yeah one could try to go that way although all such attempts I've seen (such as [1]) seem way too cumbersome, I would just build a silent PC, way simpler, more extensible and easily replaceable components. I thought you meant to use RAIN for business servers. [1] https://www.jeffgeerling.com/blog/2020/building-fastest-raspberry-pi-nas-sata-raid https://www.jeffgeerling.com/blog/2020/building-fastest-rasp...

ChuckMcM 6y ago

The more interesting data point is that the "URE" spec. That is "unrecoverable read error". All of these drives do track at a time with full ECC to facilitate reading in the presence of noise, but a track write can have effects on adjacent tracks and temperature can affect head flying height, and vibration can affect head tracking, etc. All of which, statistically, can create a non-recoverable bit error. Typically this spec is like 1 in 10^14 bits. What it interesting about this spec is that it is statistical, meaning that the more times you read the bits, the more likely you are to hit a URE. HDD manufacturers do all sorts of things to avoid them (re-trys, track mirroring, etc) but ultimately it is a mechanical system and there are a LOT of bits. When I was at NetApp, the company would get reports of UREs that were fixed by RAID reconstruction (so you read a disk, you get an error and you fix it by the ECC in the other drives). And we had determined that by 4TB it would be "stupid" to using mirroring for protection since if a drive failed you couldn't count on being able to re-silver the mirror by getting a clean read of the other half. (something like 1 in 20 attempts would fail and you'd have data loss). And of course reconstructing a RAID-5 group for a failed drive you need to read the other drives, for larger groups of disks (which people did to maximize data storage) you started becoming at risk of being able to reconstruct all of the stripes. That was the genesis of doing dual parity (which Netapp called 'diagonal parity' because of the design). The ZFS folks have also considered this and designed both dual and TRIPLE parity which makes sense for these larger drives! It already takes a long time to reconstruct any "wide" vDev in ZFS that is zraid2. So one wonders if you could solve for device capacity where 'reconstruction takes 3 years' (the depreciated life span of a drive at Google).

NateLawson 6y ago
Kirk McKusick took this kind of scale into consideration when designing softupdates for UFS. One of his criticisms of the competing log structured filesystems at the time was that the replay process would require RAM in excess of any server's capacity once disk drives were large enough. He had designed the softupdates recovery process to be incremental and use a fixed amount of RAM that didn't scale with volume size. This kind of holistic systems' awareness is a sign of software that's designed well.
effie 6y ago

Dylan16807 6y ago

You did the wrong calculation. Either you used a very pessimistic estimate for drive speed and forgot to factor in the increase in linear density, or you used a relatively accurate calculation and mixed up megabytes per second with megabits per second.

effie 6y ago
With 200MB/s I'm getting one week as the lower limit for rebuild time. If the drives are used heavily while rebuilding, this can go much longer.
- Dylan16807 6y ago
  The increase in platter density means you should expect numbers more like 600MB/s. > If the drives are used heavily while rebuilding, this can go much longer. True but that's a separate factor and depends on how the array is laid out.
  - effie 6y ago
    You are right, let's hope those 600MB/s will be the case.

Mandatum 6y ago

It took 24 hours for me to validate/check my 16TB disk (I don't know the linux commands, it was a tool /r/DataHoarders suggested). I've upgraded the quality of all my video files and I've still got TB's of space to go.. 120TB.. Man, to those in the future laughing at this comment with your 120TB games and your 60TB super-high-definition VR feature length movies.. I'm jealous.