6 ms·

With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to

With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to recovery from backup and downtime. For example, if a whole rack goes down, and they are on 32TB drives in the future, for example, could it takes a week or more for their data to get online?

andy4blaze 10y ago
The Backblaze Vault design mitigates that as the "raid array" is scattered across 20 different Storage Pods in twenty different racks. You'd need more than three racks to go down before you would be offline. Backup systems in place make that highly unlikely. Andy at Backblaze.
- fpp 10y ago
  I'll see from your reports that you're migrating 1000s of 2TB drives to 8TB drives - what is actually happening to your old 2TB drives? Guess you are not throwing those away so what are you doing with them.
  - brianwski 10y ago
    Disclaimer: I work at Backblaze. We securely wipe the drives, then we sell them to a "used hard drive reseller".
    - SteveNuts 10y ago
      I know it's a lot easier to wholesale resell them, but it'd be so awesome if you could dedicate a portion to be sold to the homelabbers among us.
      - jerguismi 10y ago
        I don't know how that would be more awesome, essentially spending effort towards the upper middle-class hobbyists. I would rather see them sell them directly to the highest bidder and focus the effort on improving their business, or give the drives to charity.
    - kikoreis 10y ago
      Does "secure wipe" mean ATA SE, i.e. hdparm --security-erase, or --security-erase-enhanced?
      - ams6110 10y ago
        For that volume I'd think a degausser would be used.
        teraflop 10y ago
        AFAIK, if you physically destroy the contents of a drive with a strong magnetic field, you also destroy the servo tracks, rendering the drive useless. Modern hard drives can't "low-level format" themselves; they're not mechanically precise enough.
        deelowe 10y ago
        Degaussers render the drive nonfunctional.
  - theandrewbailey 10y ago
    They were reusing their 1TB drives to test new pods with. But I think they would only need a hundred or so for that, unless they die faster in that workload. https://www.backblaze.com/blog/hard-drive-reliability-q4-2015/ https://www.backblaze.com/blog/hard-drive-reliability-q4-201...
- pfarnsworth 10y ago
  I guess my question is more about the long term implications of transfer speeds increasing much slower than hard drives space. And if there's a hidden risk to downtime because all of a sudden, a full rack of 32 or even 64TB drive will take day(s) to transfer as opposed to an hour or so because transfer speeds are so slow.
  - andy4blaze 10y ago
    I suppose if you had a single 32TB drive that went offline for say a week, and then once it came back online you'd have some type of pent up demand and a "slow" transfer speed. Storage systems in general spread the load across multiple devices so the effect of slow transfer speed is near zero in most backup and archiving applications and maybe more problematic in transactional applications.
  - pgeorgi 10y ago
    There are implications. ZFS' raid-z2 (which uses 2 disks for redundancy data) was pitched - in 2008 or so - with the idea that disks are now large enough that recovery can take long enough that it's likely that another disk in the raid set breaks before the redundancy is back.
dom0 10y ago
Well, uh, yes. No one really expects that rotational rust will get much faster, and in fact history shows that, compared to the increase in density, the increase in transfer rates are laughable at best. Between 1990 and today you are probably looking at a 20 000 times increase in density, yet transfer rates only increased around a factor of around 150-200. [In fact, from the early 1960 to today it's only a factor of about 1000. There are quite possibly few performance metrics that increased so slowly as disk transfer rate]. Why is that? Increasing density does only marginally increase transfer speed: Most density increases are achieved by packing more tracks onto the platter, while storing more sectors per track plays a minor role. But a single R/W head can only read a single track, not parallel tracks, hence speed only increases if you pack more sectors into each track, not by increasing the number of tracks. That's why between today and 10 years ago performance in desktop or server drives only differs a little, compared to the capacity increase to 8+ TB in a 3.5" drive. More platters also don't help in transfer rate, because the alignment of all heads on the actuator is fixed: at any given time only one platter and one R/W head is used (locked to the track). [More platters can help reduce seek time in certain scenarios though] More disks on the other hand...
- kardos 10y ago
  Is there a reason you couldn't use a second set of actuators?
  - dom0 10y ago
    There's kinda not enough space: https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Samsung_HD753LJ_03-Opened.jpg/1024px-Samsung_HD753LJ_03-Opened.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Sa... You could probably do it if the platter diameter were reduced, which would then accordingly reduce capacity as well. At that point you could just as well use two drives and also get lower overall failure probability. Or you use SSD caches, or memory caches, ...
    - kardos 10y ago
      I was thinking of a Frankendrive with 5 actuators, which can do 5 parallel reads/writes, as long as they don't overlap. But the new dimensions would probably be the roadblock.
drzaiusapelord 10y ago
Heck, just a raid rebuild takes days nowadays if you dare use RAID5 still or, less risky, RAID6. The smart move is to keep several smaller arrays instead of one big one. This lowers risk as well. I dont put in anything bigger than 7 disks into production. Past that I'm just asking for trouble. Its better to have 4 7 disk arrays than one 28 disk array. A drive fail means a quick rebuild and a restore is going to be 1/4 the time.
- cm2187 10y ago
  Stupid question but why does it take more time to rebuild a 10 disks array than a 6 disks array. I mean modern PCIe can go several GB/s and the disks are in parallel, so it should take the same time to rebuild irrespective of how many disks.
Retric 10y ago
Transfer increases as square root of drive density. http://www.anandtech.com/show/10541/western-digital-adds-helium-filled-wd-gold-10-tb-hdd-to-lineup http://www.anandtech.com/show/10541/western-digital-adds-hel... Shows 10GB drives at 249 MB/s sustained sequential transfer rate. That takes ~11 hours to fill. At 400x the density it would take 20x as long or 9 days. I don't think HDD drives are hitting 400x the density any time soon but if they did it would be a problem. However, in an array you could take a month to fill a drive to 75% without causing to much trouble. Assuming you had enough drives. That's around a ~80PB limit drive. IMO, the real issue is it would take another month to download all that data. Relegating HDD firmly into archival storage. PS: I don't think rust is going to get into those density's making this far less of an issue.
fulafel 10y ago
Internet connection speeds have been flat longer so it eves out I guess.

andy4blaze 10y ago

The Backblaze Vault design mitigates that as the "raid array" is scattered across 20 different Storage Pods in twenty different racks. You'd need more than three racks to go down before you would be offline. Backup systems in place make that highly unlikely. Andy at Backblaze.

fpp 10y ago
I'll see from your reports that you're migrating 1000s of 2TB drives to 8TB drives - what is actually happening to your old 2TB drives? Guess you are not throwing those away so what are you doing with them.
- brianwski 10y ago
  Disclaimer: I work at Backblaze. We securely wipe the drives, then we sell them to a "used hard drive reseller".
  - SteveNuts 10y ago
    I know it's a lot easier to wholesale resell them, but it'd be so awesome if you could dedicate a portion to be sold to the homelabbers among us.
    - jerguismi 10y ago
      I don't know how that would be more awesome, essentially spending effort towards the upper middle-class hobbyists. I would rather see them sell them directly to the highest bidder and focus the effort on improving their business, or give the drives to charity.
  - kikoreis 10y ago
    Does "secure wipe" mean ATA SE, i.e. hdparm --security-erase, or --security-erase-enhanced?
    - ams6110 10y ago
      For that volume I'd think a degausser would be used.
      - teraflop 10y ago
        AFAIK, if you physically destroy the contents of a drive with a strong magnetic field, you also destroy the servo tracks, rendering the drive useless. Modern hard drives can't "low-level format" themselves; they're not mechanically precise enough.
      - deelowe 10y ago
        Degaussers render the drive nonfunctional.
- theandrewbailey 10y ago
  They were reusing their 1TB drives to test new pods with. But I think they would only need a hundred or so for that, unless they die faster in that workload. https://www.backblaze.com/blog/hard-drive-reliability-q4-2015/ https://www.backblaze.com/blog/hard-drive-reliability-q4-201...
pfarnsworth 10y ago
I guess my question is more about the long term implications of transfer speeds increasing much slower than hard drives space. And if there's a hidden risk to downtime because all of a sudden, a full rack of 32 or even 64TB drive will take day(s) to transfer as opposed to an hour or so because transfer speeds are so slow.
- andy4blaze 10y ago
  I suppose if you had a single 32TB drive that went offline for say a week, and then once it came back online you'd have some type of pent up demand and a "slow" transfer speed. Storage systems in general spread the load across multiple devices so the effect of slow transfer speed is near zero in most backup and archiving applications and maybe more problematic in transactional applications.
- pgeorgi 10y ago
  There are implications. ZFS' raid-z2 (which uses 2 disks for redundancy data) was pitched - in 2008 or so - with the idea that disks are now large enough that recovery can take long enough that it's likely that another disk in the raid set breaks before the redundancy is back.

dom0 10y ago

Well, uh, yes. No one really expects that rotational rust will get much faster, and in fact history shows that, compared to the increase in density, the increase in transfer rates are laughable at best. Between 1990 and today you are probably looking at a 20 000 times increase in density, yet transfer rates only increased around a factor of around 150-200. [In fact, from the early 1960 to today it's only a factor of about 1000. There are quite possibly few performance metrics that increased so slowly as disk transfer rate]. Why is that? Increasing density does only marginally increase transfer speed: Most density increases are achieved by packing more tracks onto the platter, while storing more sectors per track plays a minor role. But a single R/W head can only read a single track, not parallel tracks, hence speed only increases if you pack more sectors into each track, not by increasing the number of tracks. That's why between today and 10 years ago performance in desktop or server drives only differs a little, compared to the capacity increase to 8+ TB in a 3.5" drive. More platters also don't help in transfer rate, because the alignment of all heads on the actuator is fixed: at any given time only one platter and one R/W head is used (locked to the track). [More platters can help reduce seek time in certain scenarios though] More disks on the other hand...

kardos 10y ago
Is there a reason you couldn't use a second set of actuators?
- dom0 10y ago
  There's kinda not enough space: https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Samsung_HD753LJ_03-Opened.jpg/1024px-Samsung_HD753LJ_03-Opened.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Sa... You could probably do it if the platter diameter were reduced, which would then accordingly reduce capacity as well. At that point you could just as well use two drives and also get lower overall failure probability. Or you use SSD caches, or memory caches, ...
  - kardos 10y ago
    I was thinking of a Frankendrive with 5 actuators, which can do 5 parallel reads/writes, as long as they don't overlap. But the new dimensions would probably be the roadblock.

drzaiusapelord 10y ago

Heck, just a raid rebuild takes days nowadays if you dare use RAID5 still or, less risky, RAID6. The smart move is to keep several smaller arrays instead of one big one. This lowers risk as well. I dont put in anything bigger than 7 disks into production. Past that I'm just asking for trouble. Its better to have 4 7 disk arrays than one 28 disk array. A drive fail means a quick rebuild and a restore is going to be 1/4 the time.

cm2187 10y ago
Stupid question but why does it take more time to rebuild a 10 disks array than a 6 disks array. I mean modern PCIe can go several GB/s and the disks are in parallel, so it should take the same time to rebuild irrespective of how many disks.

Retric 10y ago

Transfer increases as square root of drive density. http://www.anandtech.com/show/10541/western-digital-adds-helium-filled-wd-gold-10-tb-hdd-to-lineup http://www.anandtech.com/show/10541/western-digital-adds-hel... Shows 10GB drives at 249 MB/s sustained sequential transfer rate. That takes ~11 hours to fill. At 400x the density it would take 20x as long or 9 days. I don't think HDD drives are hitting 400x the density any time soon but if they did it would be a problem. However, in an array you could take a month to fill a drive to 75% without causing to much trouble. Assuming you had enough drives. That's around a ~80PB limit drive. IMO, the real issue is it would take another month to download all that data. Relegating HDD firmly into archival storage. PS: I don't think rust is going to get into those density's making this far less of an issue.

fulafel 10y ago

Internet connection speeds have been flat longer so it eves out I guess.