15 ms·

HGST gets closer to shipping 10TB HDD

89 points by tanglesome 12y ago

fiatmoney 12y ago
The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing some pretty sophisticated replication, validation, & caching.
- the8472 12y ago
  Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk? Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.
  - fiatmoney 12y ago
    Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in hour 14 of your rebuild. Of course, that's in the context of an actual RAID array (or ZFS pool, etc.) - if you've got a replication strategy that spreads multiple entire copies over multiple systems, you're less vulnerable and these kinds of disks are a good choice.
    - the8472 12y ago
      Well, that sounds more like worrying about a single point of failure (e.g. that power source that can take out several drives at once) than what the redundancy is designed to protect against, i.e. uncorrelated failures of one of the drives in the array. Of course having no redundancy at all during a rebuild that takes more than a negligible amount of time is still worrisome, but that's why we have multiple-redundancy raid schemes. But you still have to untangle your concerns. A single point (or cause) of failure has to be handled differently than increasing probabilities of encountering independent UREs on separate drives during rebuild.
- MPSimmons 12y ago
  Shingled drives are probably more likely used in a RAID-10 scenario where you're not recalculating parity, but just rebuilding linearly. These drives (like the Western Digital Purple) are used predominantly for archival data. You should never have to randomly write to, for instance, surveillance video, or similar streamed media. This isn't for your SQL database server.
  - mtanski 12y ago
    This is pedantic, but unless the filesystem is build specifically for WORM devices you always end up with some non-sequential data. In the use cases you're describing they'll be some filesystem metadata (freemaps, directory entries, inode tables) that will need to be rewritten.
- hyperion2010 12y ago
  Can't this problem be partially solved by having more read/write heads? I don't think it will fit in the same form factor, but if the limit is how fast you can spin the disk then having multiple independently controlled read/write heads per platter seems like one way out of them problem.
  - morcheeba 12y ago
    seagate dual head design: http://www.tomshardware.com/news/seagate-hdd-harddrive,8279.html http://www.tomshardware.com/news/seagate-hdd-harddrive,8279....
- moe 12y ago
  the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.
  - ghshephard 12y ago
    Unless the root cause is something to do with the array - flakey controller, or power supply.
    - craigyk 12y ago
      these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?
      - DiabloD3 12y ago
        ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home. And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane. Once you try ZFS, you never go back.
  - vidarh 12y ago
    Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller. (We didn't lose any data - everything is replicated elsewhere in addition to the raid)
  - 12y ago
- ghshephard 12y ago
  This is probably why people like BackBlaze use RAIN (Redundant Array of Inexpensive Nodes) instead of RAID. Also, their drive failure is 10 disks/day out of a population of 44,100 drives, about 0.02%. So, between the lack of correlation of failure (their Nodes, or, "Pods" are in different racks), their ability to lose an entire POD regardless (they have 3 parity pods), and the relatively low disk failure rate - the large drives aren't a problem. A rebuild in 2-3 days is more than fine.
  - sitkack 12y ago
    There is a very interesting engineering optimization strategy at play here, would love to do/see a blog post on it. Simple structures, with a large safety margin and simple control flow vs complex structures, very light weight but at the limits of capacity. Like an ethernet switch doesn't need complex flow routing algorithms when the raw backplane bandwidth is some multiplier higher than will ever experience contention. It always seems to me, that things built near their design limits will eventually expose a catastrophic flaw. Human engineered items that have stood the test of time, Roman Aqueducts, Brooklyn Bridge, DC3, Dodge Dart, Toyota Corolla, AK47, PDP8 all have commonalities in their design philosophies.
- eck 12y ago
  > the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.
  - ghshephard 12y ago
    I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.
    - ryan-c 12y ago
      That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...
    - cmurf 12y ago
      The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.
- epistasis 12y ago
  Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)
  - ghshephard 12y ago
    A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?
    - chrisbolt 12y ago
      > Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object? Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc). http://ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types http://ceph.com/docs/master/rados/operations/crush-map/#crus...
    - epistasis 12y ago
      Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week. And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.
- agumonkey 12y ago
  I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place. [1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-2x-faster/ http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...
  - cdr 12y ago
    Apple must not be using the top-tier Samsung M.2 SSD; they can write at 1500MB/s+ and read at 2100MB/s. 1024MB/s is nothing. Now that SSDs are moving to using PCI-E lanes instead of SATA, we'll probably see quite a speed race. Really looking forward to the Intel 750, which will straight up be a PCI-E card for the desktop version. http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-ssd,4045.html http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-s...
- pkaye 12y ago
  I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.
  - ars 12y ago
    http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC78288257D8F0004C55C/$file/HGST-Delivers-Rebuild-Assist-Mode-for-Faster-RAID-Recovery-WP23.pdf http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC7... The way it works is by realizing there are only a few bad sectors on the failed drive, so rebuild that area first, the used the "bad" drive to help rebuild the rest of the array at full speed.
    - rasz_pl 12y ago
      Sounds clever. Too bad SSD (due to being shit and simply dying all at once) will never support that.
      - dmm 12y ago
        Spinning rust can fail all at once too. When ssds die of write endurance they loudly complain long before they actually lose data.
        ars 12y ago
        It can, sure, but I think most failures are of the slower sort. And SSD's only complain if you listen. Does Windows come with a smart checking tool by default? Don't you have to manually install one and check? I suspect most people with an SSD have no idea where it's holding. But it doesn't change the "defective by design" nature. It's one thing if you can't do anything about it - but the bricking design is not good. It should go read only on max writes, not brick.
        pkaye 12y ago
        "And SSD's only complain if you listen." I'm not sure how this is different from HDDs? For both of them you have to listen and install a SMART checking tool. And HDD failure rates are not great either ranging from 2-12% according to recent Backblaze data.
- letstryagain 12y ago
  We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1) The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.
  - jacquesm 12y ago
    That creates a substantial window during which you could lose the second drive.
    - gtwy 12y ago
      I'm sure they have their reasons (I won't pretend to know every detail about their setup) but it's not a solution I would adopt in our environment.
    - letstryagain 12y ago
      RAID 1 rebuild time is the same as RAID 10 but if the entire thing fails you only lose one drive's info, not the entire RAID 10 array.
- Viper007Bond 12y ago
  That seems slow. I have a 6 x 6TB RAID 5 array (27TB) for non-critical data storage and it takes less something like 22 hours to do its weekly parity check. I'm running firmware RAID via an onboard Intel controller on an ASUS board.
ars 12y ago
Wow, hard drives are getting complicated! You have to write 256MB to change a single bit? And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone? You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.
- the8472 12y ago
  > You would need some sort of battery backed scratch/cache area to pull this off Or a copy-on-write filesystem. With some tweaking they can essentially fill one zone by linearly writing new copies (and metadata) to it while garbage-collecting (copying GC, with compactation) old zones in the background
- ryan-c 12y ago
  Having maybe 16 to 64 GB of flash storage (for writeback cache) would probably make a significant difference here. I think over the next five to ten years tiered storage will probably go mainstream.
  - mtanski 12y ago
    Tiered storage is already here. If you're using block storage in the cloud those guys are already do all sorts of optimizations (for their costs) that include tiering. And on your own desktop / own server side you can do this today on Linux with bcache or dm-cache. Works great for databases.
- gamegoblin 12y ago
  You only need to rewrite the whole zone if you want to modify something that's before the write pointer. That is, you can write data sequentially without penalty. It's modifying existing data that's the issue.
kondro 12y ago
Before people make too many comments about the complexity and speed of these drives, it is important to realise that these drives are designed for append-mostly archival of data.
deleted 12y ago
[deleted]
toolslive 12y ago
One of the problem with these HDDs is that you still have the same number of spindles. These spindles now need to cover a lot more data, and they were already `challenged`
qt_scientist 12y ago
Yup, RAID is being replaced by object storage for large scale systems. Last week, HGST acquired object storage startup Amplidata...
mtanski 12y ago
We spent a bit of time talking about about these "shingled drives" at the LSF/MM summit earlier in the week in Boston. Storage is becoming a more fascinating topic with so many technologies (SSDs, shingled disk, non-volatile memory) and so new open source solutions (Ceph, Luster).
atYevP 12y ago
Yev from Backblaze here -> gimme gimme gimme gimme!

89 points by tanglesome 12y ago

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing some pretty sophisticated replication, validation, & caching.

the8472 12y ago
Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk? Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.
- fiatmoney 12y ago
  Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in hour 14 of your rebuild. Of course, that's in the context of an actual RAID array (or ZFS pool, etc.) - if you've got a replication strategy that spreads multiple entire copies over multiple systems, you're less vulnerable and these kinds of disks are a good choice.
  - the8472 12y ago
    Well, that sounds more like worrying about a single point of failure (e.g. that power source that can take out several drives at once) than what the redundancy is designed to protect against, i.e. uncorrelated failures of one of the drives in the array. Of course having no redundancy at all during a rebuild that takes more than a negligible amount of time is still worrisome, but that's why we have multiple-redundancy raid schemes. But you still have to untangle your concerns. A single point (or cause) of failure has to be handled differently than increasing probabilities of encountering independent UREs on separate drives during rebuild.
MPSimmons 12y ago
Shingled drives are probably more likely used in a RAID-10 scenario where you're not recalculating parity, but just rebuilding linearly. These drives (like the Western Digital Purple) are used predominantly for archival data. You should never have to randomly write to, for instance, surveillance video, or similar streamed media. This isn't for your SQL database server.
- mtanski 12y ago
  This is pedantic, but unless the filesystem is build specifically for WORM devices you always end up with some non-sequential data. In the use cases you're describing they'll be some filesystem metadata (freemaps, directory entries, inode tables) that will need to be rewritten.
hyperion2010 12y ago
Can't this problem be partially solved by having more read/write heads? I don't think it will fit in the same form factor, but if the limit is how fast you can spin the disk then having multiple independently controlled read/write heads per platter seems like one way out of them problem.
- morcheeba 12y ago
  seagate dual head design: http://www.tomshardware.com/news/seagate-hdd-harddrive,8279.html http://www.tomshardware.com/news/seagate-hdd-harddrive,8279....
moe 12y ago
the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Two disk failures in the same array within 15 hours is very rare.
- ghshephard 12y ago
  Unless the root cause is something to do with the array - flakey controller, or power supply.
  - craigyk 12y ago
    these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?
    - DiabloD3 12y ago
      ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home. And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane. Once you try ZFS, you never go back.
- vidarh 12y ago
  Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller. (We didn't lose any data - everything is replicated elsewhere in addition to the raid)
- 12y ago
ghshephard 12y ago
This is probably why people like BackBlaze use RAIN (Redundant Array of Inexpensive Nodes) instead of RAID. Also, their drive failure is 10 disks/day out of a population of 44,100 drives, about 0.02%. So, between the lack of correlation of failure (their Nodes, or, "Pods" are in different racks), their ability to lose an entire POD regardless (they have 3 parity pods), and the relatively low disk failure rate - the large drives aren't a problem. A rebuild in 2-3 days is more than fine.
- sitkack 12y ago
  There is a very interesting engineering optimization strategy at play here, would love to do/see a blog post on it. Simple structures, with a large safety margin and simple control flow vs complex structures, very light weight but at the limits of capacity. Like an ethernet switch doesn't need complex flow routing algorithms when the raw backplane bandwidth is some multiplier higher than will ever experience contention. It always seems to me, that things built near their design limits will eventually expose a catastrophic flaw. Human engineered items that have stood the test of time, Roman Aqueducts, Brooklyn Bridge, DC3, Dodge Dart, Toyota Corolla, AK47, PDP8 all have commonalities in their design philosophies.
eck 12y ago
> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high. Well there's one issue that another whole drive will fail and you're screwed. The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.
- ghshephard 12y ago
  I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.
  - ryan-c 12y ago
    That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively. http://www.enterprisestorageforum.com/storage-hardware/selecting-a-disk-drive-how-not-to-do-research-1.html http://www.enterprisestorageforum.com/storage-hardware/selec...
  - cmurf 12y ago
    The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.
epistasis 12y ago
Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)
- ghshephard 12y ago
  A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?
  - chrisbolt 12y ago
    > Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object? Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc). http://ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types http://ceph.com/docs/master/rados/operations/crush-map/#crus...
  - epistasis 12y ago
    Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week. And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.
agumonkey 12y ago
I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place. [1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-2x-faster/ http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...
- cdr 12y ago
  Apple must not be using the top-tier Samsung M.2 SSD; they can write at 1500MB/s+ and read at 2100MB/s. 1024MB/s is nothing. Now that SSDs are moving to using PCI-E lanes instead of SATA, we'll probably see quite a speed race. Really looking forward to the Intel 750, which will straight up be a PCI-E card for the desktop version. http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-ssd,4045.html http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-s...
pkaye 12y ago
I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.
- ars 12y ago
  http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC78288257D8F0004C55C/$file/HGST-Delivers-Rebuild-Assist-Mode-for-Faster-RAID-Recovery-WP23.pdf http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC7... The way it works is by realizing there are only a few bad sectors on the failed drive, so rebuild that area first, the used the "bad" drive to help rebuild the rest of the array at full speed.
  - rasz_pl 12y ago
    Sounds clever. Too bad SSD (due to being shit and simply dying all at once) will never support that.
    - dmm 12y ago
      Spinning rust can fail all at once too. When ssds die of write endurance they loudly complain long before they actually lose data.
      - ars 12y ago
        It can, sure, but I think most failures are of the slower sort. And SSD's only complain if you listen. Does Windows come with a smart checking tool by default? Don't you have to manually install one and check? I suspect most people with an SSD have no idea where it's holding. But it doesn't change the "defective by design" nature. It's one thing if you can't do anything about it - but the bricking design is not good. It should go read only on max writes, not brick.
        pkaye 12y ago
        "And SSD's only complain if you listen." I'm not sure how this is different from HDDs? For both of them you have to listen and install a SMART checking tool. And HDD failure rates are not great either ranging from 2-12% according to recent Backblaze data.
letstryagain 12y ago
We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1) The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.
- jacquesm 12y ago
  That creates a substantial window during which you could lose the second drive.
  - gtwy 12y ago
    I'm sure they have their reasons (I won't pretend to know every detail about their setup) but it's not a solution I would adopt in our environment.
  - letstryagain 12y ago
    RAID 1 rebuild time is the same as RAID 10 but if the entire thing fails you only lose one drive's info, not the entire RAID 10 array.
Viper007Bond 12y ago
That seems slow. I have a 6 x 6TB RAID 5 array (27TB) for non-critical data storage and it takes less something like 22 hours to do its weekly parity check. I'm running firmware RAID via an onboard Intel controller on an ASUS board.

ars 12y ago

Wow, hard drives are getting complicated! You have to write 256MB to change a single bit? And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone? You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.

the8472 12y ago
> You would need some sort of battery backed scratch/cache area to pull this off Or a copy-on-write filesystem. With some tweaking they can essentially fill one zone by linearly writing new copies (and metadata) to it while garbage-collecting (copying GC, with compactation) old zones in the background
ryan-c 12y ago
Having maybe 16 to 64 GB of flash storage (for writeback cache) would probably make a significant difference here. I think over the next five to ten years tiered storage will probably go mainstream.
- mtanski 12y ago
  Tiered storage is already here. If you're using block storage in the cloud those guys are already do all sorts of optimizations (for their costs) that include tiering. And on your own desktop / own server side you can do this today on Linux with bcache or dm-cache. Works great for databases.
gamegoblin 12y ago
You only need to rewrite the whole zone if you want to modify something that's before the write pointer. That is, you can write data sequentially without penalty. It's modifying existing data that's the issue.

kondro 12y ago

Before people make too many comments about the complexity and speed of these drives, it is important to realise that these drives are designed for append-mostly archival of data.

deleted 12y ago

[deleted]

toolslive 12y ago

One of the problem with these HDDs is that you still have the same number of spindles. These spindles now need to cover a lot more data, and they were already `challenged`

qt_scientist 12y ago

Yup, RAID is being replaced by object storage for large scale systems. Last week, HGST acquired object storage startup Amplidata...

mtanski 12y ago

We spent a bit of time talking about about these "shingled drives" at the LSF/MM summit earlier in the week in Boston. Storage is becoming a more fascinating topic with so many technologies (SSDs, shingled disk, non-volatile memory) and so new open source solutions (Ceph, Luster).

atYevP 12y ago

Yev from Backblaze here -> gimme gimme gimme gimme!