14 ms·

Google seeks new disks for data centers

160 points by dbcooper 11y ago

tallanvor 11y ago
That blog post doesn't really say anything. You have to click through a couple of links to find the real paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44830.pdf https://static.googleusercontent.com/media/research.google.c... which has some interesting ideas, although it's a bit too brief for the subject, I think.
ck2 11y ago
Should be interesting to see what they come up with. Somehow I don't think they will find things better than multi-million dollar research like HAMR for write-once read-many, but we'll see I guess. I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? Why not go back to 8 inch for massive surface area? Or is that too much mass to spin.
- withjive 11y ago
  Indeed, it will be interesting to see what Google can think up compared to the other big researchers. Though I wonder if the goals are slightly different, as Google is focusing on "Cloud Storage", rather than single hard performance. Which as they mentioned, the Cloud doesn't need to have great reliability (since data is assumed to be already redundant), this may allow them to decrease the cost of hard drives for exchange in cheaper disks that may fail more. Doesn't sound like they are expecting to increase size/performance alone with this effort.
- StreamBright 11y ago
  In the long run anything that moves in DC is way less reliable than systems without moving parts. On the top of that, these systems also need more energy to maintain the movement. If you increase the size than you need better materials to keep up with the increased mechanical load. I bet by 2020 we are going to have better ways of storing large amount of data than a spinning disk. The new optical drive would be a good candidate for some use cases, mostly for archiving though: http://www.sciencealert.com/this-new-5d-data-storage-disc-can-store-360tb-of-data-for-14-billion-years http://www.sciencealert.com/this-new-5d-data-storage-disc-ca... For continuous read/write workloads SSDs are the best for many reasons (energy efficiency, latency, IO bandwidth) and I think the world is moving that direction to replace spinning disks with these. I am hoping to see a bump in the SSD capacity as well. We are here now: http://www.extremetech.com/computing/221303-the-worlds-biggest-ssd-has-arrived-at-13tb http://www.extremetech.com/computing/221303-the-worlds-bigge...
  - ucaetano 11y ago
    From the article: "An obvious question is why are we talking about spinning disks at all, rather than SSDs, which have higher IOPS and are the “future” of storage." "The root reason is that the cost per GB remains too high, and more importantly that the growth rates in capacity/$ between disks and SSDs are relatively close (at least for SSDs that have sufficient numbers of program-erase cycles to use in data centers), so that cost will not change enough in the coming decade."
    - zaroth 11y ago
      Well at least for YouTube video storage, I imagine the P/E cycles are not a problem. There have been issues with longevity like on the Samsung 840 EVO drives, but if you can avoid that then you can build some incredibly dense, fast, reliable and power efficient storage on top of V-NAND. Cost per GB for SSDs is definitely catching up to HDD [1] and I think is expected to match or beat HDD within the decade. Power efficiency should be better on SSD and MTBF will be much better than HDD, so I would have thought TCO for SSD would have HDDs basically beat already. [1] https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-hdd-chart-retry-100631270-orig.png https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-...
- cm2187 11y ago
  For 100TB disks to make sense the read/write speed would have to improve considerably, otherwise the disk will be useless other than for cold storage. I haven't really seen spinning disks speed improving much. It would take weeks to rebuild an array of 100TB disks at the current speed.
  - rbanffy 11y ago
    I remember a washing machine sized IBM mainframe hard disk with tho sets of heads. That's one trivial way to double bandwidth and cut latency (it'll read like a RAID-1 and write as a RAID-0). Also, the more platters, the more data you can read on a single rotation.
    - cm2187 11y ago
      You double bandwidth for a single access. But if you are storing 100TB on a single drive, it's unlikely to be 100TB related to a single user. You will want to make a lot of requests to this data simultaneously, a lot more requests than what you would to a disk containing only 1TB of data.
  - reitzensteinm 11y ago
    100TB should only be 5x (i.e. sqrt( 25 )) slower to access each byte sequentially than a 4TB drive (although shingling will screw up this calculation, since it provides more tracks, but not more density on each track). Which is a long time, but not totally out of bounds for the way we're currently using disks. The coming decade is going to be very interesting, SSDs are pushing forward on an insane trajectory, which could either fall apart as the benefit of node shrinks disappear, or accelerate through a virtuous cycle as unit volume increases. It's even plausible that spinning rust is on death's door in that time frame. The price floor of SSDs is much lower than that of hard drives (i.e. how much does an 8gb hard drive cost to manufacture today in volume? As much as a 500 gb drive), meaning that both mildly performance conscious and price sensitive consumers both move entirely over. Enterprise customers will still chase the price/gb and storage density of traditional hard drives, but without the consumer dollars to invest in R&D, that could start to stall, quickly leaving them with diminishing advantage and obsolete in the blink of an eye.
- darkr 11y ago
  > I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise). Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure
  - SixSigma 11y ago
    > you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.
  - __david__ 11y ago
    > Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.
    - ck2 11y ago
      Is that defaulting to EXT4 or XFS ?
- simplemath 11y ago
  If they go with anything Arm and Platter I'd be shocked. I'm imagining a 10 acre array of NAND
  - baruch 11y ago
    They are saying the paper they want arms and platters as it's more economical. An HDD is cheaper than an SSD of the same capacity at four to ten times and I don't think the electricity cost will offset this if you care more about capacity than performance.
    - rbanffy 11y ago
      It is right now, but silicon and precision mechanics scale with different curves. There is a point where spinning metal will no longer be viable as a competitor and I'll guess, based on a hunch, it's not that far into the future.
      - tytso 11y ago
        Flash is not getting cheaper faster than HDD's, once you control for write endurance. One of the big messages in the paper is that IOPS matter. If you only have a petabyte or two of data, you can use a brute force solution such as flash or 15k drives. But at Cloud scale (we used to say Google scale, but there other companies like Amazon and Microsoft and Facebook that run at these scales now) you just can't afford that much flash. So the trick is to use every last IOPS that you can out of each and every disk, and that means provisioning and spreading out your data across an entire fleet of disks. The alternative, where you silo your storage in disk pools, and where some disks or disk pools have IOPS that are unused --- wasted, and where you make up for that by using flash for your hot and warm workloads is just a much more expensive way to do things. It's for that reason that SMR drives aren't all that interesting. Sure, you get 20% more capacity, but at the cost of burning most of your IOPS for GC. Or if you move all of your SMR-friendly data to the SMR drives, that's cold data, and it means that you've wasted all of the IOPS that those SMR drives are capable of. Which means the rest of your disk fleet is now much hotter, and you might have to buy more CMR spindles to cover the cost --- at which point the cost advantage of SMR drives disappears. (I helped to author the section about SMR drives, and why we are interested in hybrid SMR/CMR drives as a solution to this problem; read the paper for more details.)
- Cthulhu_ 11y ago
  I for one am confident that SSD and its technology will be much more scalable and future-proof than spinning disks; stackable chips are a thing already. I'm sure that by 2020, there will be companies that can deliver a storage platform with petabytes of space in a 1U package. Just imagine one of those server things crammed full of ssd chips.
  - baruch 11y ago
    In the long run that may be true, but the same things they talk about (minus the mechanical head changes) will still hold in SSDs. The background activity is even worse for SSDs and the tail latency still very much exists. Essentially the problem is that the HDDs and SSDs are built as black boxes and are not vertically integrated with a data center storage scheme. The major win will be when the HDDs and SSDs are slightly less self-sufficient and let the higher components in the stack manage them more in depth. Being able to tell the disk "do not bother with ecc for now" and try to get the data from your RAID and if that doesnt work come back to the disk and tell it "we really really want the data, spend as much as needed on this" is very strong in both getting the tail latency down and getting the reliability to stay at the same level. For SSDs there is an added complication of garbage collection and the incredible amounts of over-provisioning needed to get consistent performance from them (at least 30% in an ME drive, the RI drives suck at consistent performance). If those can be reduced by more vertical integration that would reduce costs of the systems and make for better overall performance as well.
  - jobu 11y ago
    It amazes me that SSD drives still (mostly) conform to the same drive sizes as spinning disks. My most recent computer build used an M.2 slot SSD, and plugging that tiny little card into the motherboard seemed like the future. No cables required, and it was even smaller than the RAM sticks.
- pbhjpbhj 11y ago
  I'd imagine it's more likely to be the engineering tolerances required to keep the vertical movement to a minimum at double the radius? But I have no real insight, just a guess. It does strike me though - why do disks do serial reads, couldn't the head read several tracks at once? Why not have several heads angularly spaced around the drive? Presumably such things have been tried and found lacking in some respect. Looks like I need to do some research ...
  - eru 11y ago
    There were some disks with two heads once. They were mechanically too complicated.
  - avian 11y ago
    > couldn't the head read several tracks at once? The paper in fact suggests this and also cites an attempt by Conner to introduce multiple actuator drives: http://www.pcguide.com/ref/hdd/op/actMultiple-c.html http://www.pcguide.com/ref/hdd/op/actMultiple-c.html
- noahdesu 11y ago
  On Tuesday Eric Brewer gave a talk on this white paper at FAST. In the talk (and probably discussed more in the paper) he proposed looking at alternative disk form factors. He seemed to be particularly interested in "tall" disks, say 2U, that had the same surface area as today, but with more platters, and potentially more heads.
  - Sanddancer 11y ago
    Ah, going back to the beginning of micros, where drives were often full-height things that could easily break feet when dropped. The thing that worries me there is that having had a big full-height 9 gb seagate scsi drive back in the day is that you needed to give the drive about a hundred watts on its own to run it, with a spool up that sounded like a jet engine was about to take off. I know that motors have gotten faster since then, and platters have gotten lighter, but that's still quite a bit of mass to get up to speed. The thing that interests me more is the idea of more read/write heads. Add a second, independent set of heads for read/write and you will decrease latency a good amount. Or, a second dependent set of heads which would not provide as much of a performance boost, but would still improve latency a good amount.
- jabl 11y ago
  > Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? > Why not go back to 8 inch for massive surface area? Or is that too much mass to spin. One problem AFAIK is that larger disks wobble too much, more the faster they spin. Which is why if you break open a 3.5" 15k drive you'll see that the platters are more like 2.5", and a lot of wasted space. For 7.2k you can use bigger platters, but how much bigger? I doubt 8" is feasible. Of course you can reduce the rpm further, but that has downsides too.
- gshubert17 11y ago
  Current disk densities are about 10^12 bits/square inch. To get 800 terabits on a single platter, it would need a diameter of about 32 inches. At 10^14 bits/square inch, only 3.2 inches. Assuming a compound annual growth rate of 25%, densities will be 10-fold larger in 10 years, and 100-fold larger in 20 years. (This actually happened in hard disks between 1972 and 1992.) But that would take us to 2035, not 2020. At those densities, a bit will have to be stored inside a square about 7 or 8 nanometers on a side. I don't know whether that's possible.
baruch 11y ago
A big thing they are saying in the paper is that they want to see data center HDDs that care slightly less about their medium error rates and opt to give better areal density and more consistent tail latencies and let the system above it to handle the reliability as it needs to do anyway. I've been thinking like this for a while and would tend to reduce the ERC (Error Recovery Control) to a minimum but the disks are still not designed to have these at very low numbers and Google has several interesting ideas in their paper along these lines. It would be really awesome if they get the HDD makers to go along with this.
davidiach 11y ago
"For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage every day or about 100x the Library of Congress. As shown in the graph, this continues to grow exponentially, with a 10x increase every five years." This blew my mind. What does this even mean in terms of logistics. How many people do you need to have just to add all those hard drives? How many new datacenters do you need to build every 5 years?
- nostrademons 11y ago
  It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course, that's just YouTube, and Google has many other needs for data. But people forget just how big the denominators are on this quantity, and how effective Kryder's Law has been. They also forget how much reserve capacity there is in human labor; Google's datacenters have tiny employee counts because they are so automated, and could easily scale up into the exabyte/day range. A more interesting question is what the differential rates of Kryder's Law vs. Moore's Law will do to how we architect software. Already, people in the know say that "disk is the new tape" - disk drive capacity has been increasing much faster than seek times, bus bandwidth, and available processing power, which means that you have to start treating the drive as a sequential storage device and not as a random-access platter. That's behind a lot of the shift from B-trees (as in conventional RDBMSes) to LSM-trees (as in BigTable/LevelDB), and also the resurgence of batch-processing frameworks like MapReduce. How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive?
  - msellout 11y ago
    > How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive? I thought we were already in that situation. Cache is king.
    - 11y ago
deleted 11y ago
[deleted]
- deleted 11y ago
  [deleted]
deleted 11y ago
[deleted]
ibmthrowaway271 11y ago
> For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage every day or about 100x the Library of Congress Hmm, something up with the sums in the middle of that. 400 hours of video every minute is much more than one gigabyte per hour. It's way more than one terabyte per hour. Working backwards:- 1 PB/day =~ 42 TB/hour =~ 728 GB/minute
- lcpriest 11y ago
  I believe they are saying that the 400 hours they upload every minute are 400GB worth of data.
  - ibmthrowaway271 11y ago
    Ah, yes, good point.
- fluxquanta 11y ago
  Variable compression and video resolution?
- ucha 11y ago
  400 hours video/min = 400 GB/min = 400 x 60 x 24 GB/day = 0.576 PB/day. Close enough.
- jfasi 11y ago
  That's only if you're storing one copy of the video.
novaleaf 11y ago
reminds me of the 5.25" Quantum Bigfoot. I'd be happy with a big disk like that in a storage network / raid. https://en.wikipedia.org/wiki/Quantum_Bigfoot https://en.wikipedia.org/wiki/Quantum_Bigfoot
- mapmap 11y ago
  The article doesn't go into much detail on the performance. I'm curious if reading was any faster because of the relatively faster spin speed at the outer rings of a larger disk? Or if it was a wash with seek times increasing due to the larger area?
- dzhiurgis 11y ago
  Which reminds me a teardown EEVblog did on 1980s 11" mainframe drive: https://www.youtube.com/watch?v=CBjoWMA5d84 https://www.youtube.com/watch?v=CBjoWMA5d84
cfcef 11y ago
The point about ECC and accepting lower error rates makes a lot of sense. It's the end-to-end principle again: since all of these HDDs are going into a global pool in which each of them is disposable and written content is protected by FEC spread across multiple drives, there is no need for each drive to spend a lot of resources, going deep into diminishing returns, trying to make itself as resilient as possible. If a network transmission fails, it is retried and doesn't need to be swathed in huge numbers of elaborate checksums and ultra-reliable links; if a hard drive fails, the content is recovered from the FEC and re-written out to a new drive.

160 points by dbcooper 11y ago

That blog post doesn't really say anything. You have to click through a couple of links to find the real paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44830.pdf https://static.googleusercontent.com/media/research.google.c... which has some interesting ideas, although it's a bit too brief for the subject, I think.

ck2 11y ago

Should be interesting to see what they come up with. Somehow I don't think they will find things better than multi-million dollar research like HAMR for write-once read-many, but we'll see I guess. I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? Why not go back to 8 inch for massive surface area? Or is that too much mass to spin.

withjive 11y ago
Indeed, it will be interesting to see what Google can think up compared to the other big researchers. Though I wonder if the goals are slightly different, as Google is focusing on "Cloud Storage", rather than single hard performance. Which as they mentioned, the Cloud doesn't need to have great reliability (since data is assumed to be already redundant), this may allow them to decrease the cost of hard drives for exchange in cheaper disks that may fail more. Doesn't sound like they are expecting to increase size/performance alone with this effort.
StreamBright 11y ago
In the long run anything that moves in DC is way less reliable than systems without moving parts. On the top of that, these systems also need more energy to maintain the movement. If you increase the size than you need better materials to keep up with the increased mechanical load. I bet by 2020 we are going to have better ways of storing large amount of data than a spinning disk. The new optical drive would be a good candidate for some use cases, mostly for archiving though: http://www.sciencealert.com/this-new-5d-data-storage-disc-can-store-360tb-of-data-for-14-billion-years http://www.sciencealert.com/this-new-5d-data-storage-disc-ca... For continuous read/write workloads SSDs are the best for many reasons (energy efficiency, latency, IO bandwidth) and I think the world is moving that direction to replace spinning disks with these. I am hoping to see a bump in the SSD capacity as well. We are here now: http://www.extremetech.com/computing/221303-the-worlds-biggest-ssd-has-arrived-at-13tb http://www.extremetech.com/computing/221303-the-worlds-bigge...
- ucaetano 11y ago
  From the article: "An obvious question is why are we talking about spinning disks at all, rather than SSDs, which have higher IOPS and are the “future” of storage." "The root reason is that the cost per GB remains too high, and more importantly that the growth rates in capacity/$ between disks and SSDs are relatively close (at least for SSDs that have sufficient numbers of program-erase cycles to use in data centers), so that cost will not change enough in the coming decade."
  - zaroth 11y ago
    Well at least for YouTube video storage, I imagine the P/E cycles are not a problem. There have been issues with longevity like on the Samsung 840 EVO drives, but if you can avoid that then you can build some incredibly dense, fast, reliable and power efficient storage on top of V-NAND. Cost per GB for SSDs is definitely catching up to HDD [1] and I think is expected to match or beat HDD within the decade. Power efficiency should be better on SSD and MTBF will be much better than HDD, so I would have thought TCO for SSD would have HDDs basically beat already. [1] https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-hdd-chart-retry-100631270-orig.png https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-...
cm2187 11y ago
For 100TB disks to make sense the read/write speed would have to improve considerably, otherwise the disk will be useless other than for cold storage. I haven't really seen spinning disks speed improving much. It would take weeks to rebuild an array of 100TB disks at the current speed.
- rbanffy 11y ago
  I remember a washing machine sized IBM mainframe hard disk with tho sets of heads. That's one trivial way to double bandwidth and cut latency (it'll read like a RAID-1 and write as a RAID-0). Also, the more platters, the more data you can read on a single rotation.
  - cm2187 11y ago
    You double bandwidth for a single access. But if you are storing 100TB on a single drive, it's unlikely to be 100TB related to a single user. You will want to make a lot of requests to this data simultaneously, a lot more requests than what you would to a disk containing only 1TB of data.
- reitzensteinm 11y ago
  100TB should only be 5x (i.e. sqrt( 25 )) slower to access each byte sequentially than a 4TB drive (although shingling will screw up this calculation, since it provides more tracks, but not more density on each track). Which is a long time, but not totally out of bounds for the way we're currently using disks. The coming decade is going to be very interesting, SSDs are pushing forward on an insane trajectory, which could either fall apart as the benefit of node shrinks disappear, or accelerate through a virtuous cycle as unit volume increases. It's even plausible that spinning rust is on death's door in that time frame. The price floor of SSDs is much lower than that of hard drives (i.e. how much does an 8gb hard drive cost to manufacture today in volume? As much as a 500 gb drive), meaning that both mildly performance conscious and price sensitive consumers both move entirely over. Enterprise customers will still chase the price/gb and storage density of traditional hard drives, but without the consumer dollars to invest in R&D, that could start to stall, quickly leaving them with diminishing advantage and obsolete in the blink of an eye.
darkr 11y ago
> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise). Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure
- SixSigma 11y ago
  > you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.
- __david__ 11y ago
  > Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.
  - ck2 11y ago
    Is that defaulting to EXT4 or XFS ?
simplemath 11y ago
If they go with anything Arm and Platter I'd be shocked. I'm imagining a 10 acre array of NAND
- baruch 11y ago
  They are saying the paper they want arms and platters as it's more economical. An HDD is cheaper than an SSD of the same capacity at four to ten times and I don't think the electricity cost will offset this if you care more about capacity than performance.
  - rbanffy 11y ago
    It is right now, but silicon and precision mechanics scale with different curves. There is a point where spinning metal will no longer be viable as a competitor and I'll guess, based on a hunch, it's not that far into the future.
    - tytso 11y ago
      Flash is not getting cheaper faster than HDD's, once you control for write endurance. One of the big messages in the paper is that IOPS matter. If you only have a petabyte or two of data, you can use a brute force solution such as flash or 15k drives. But at Cloud scale (we used to say Google scale, but there other companies like Amazon and Microsoft and Facebook that run at these scales now) you just can't afford that much flash. So the trick is to use every last IOPS that you can out of each and every disk, and that means provisioning and spreading out your data across an entire fleet of disks. The alternative, where you silo your storage in disk pools, and where some disks or disk pools have IOPS that are unused --- wasted, and where you make up for that by using flash for your hot and warm workloads is just a much more expensive way to do things. It's for that reason that SMR drives aren't all that interesting. Sure, you get 20% more capacity, but at the cost of burning most of your IOPS for GC. Or if you move all of your SMR-friendly data to the SMR drives, that's cold data, and it means that you've wasted all of the IOPS that those SMR drives are capable of. Which means the rest of your disk fleet is now much hotter, and you might have to buy more CMR spindles to cover the cost --- at which point the cost advantage of SMR drives disappears. (I helped to author the section about SMR drives, and why we are interested in hybrid SMR/CMR drives as a solution to this problem; read the paper for more details.)
Cthulhu_ 11y ago
I for one am confident that SSD and its technology will be much more scalable and future-proof than spinning disks; stackable chips are a thing already. I'm sure that by 2020, there will be companies that can deliver a storage platform with petabytes of space in a 1U package. Just imagine one of those server things crammed full of ssd chips.
- baruch 11y ago
  In the long run that may be true, but the same things they talk about (minus the mechanical head changes) will still hold in SSDs. The background activity is even worse for SSDs and the tail latency still very much exists. Essentially the problem is that the HDDs and SSDs are built as black boxes and are not vertically integrated with a data center storage scheme. The major win will be when the HDDs and SSDs are slightly less self-sufficient and let the higher components in the stack manage them more in depth. Being able to tell the disk "do not bother with ecc for now" and try to get the data from your RAID and if that doesnt work come back to the disk and tell it "we really really want the data, spend as much as needed on this" is very strong in both getting the tail latency down and getting the reliability to stay at the same level. For SSDs there is an added complication of garbage collection and the incredible amounts of over-provisioning needed to get consistent performance from them (at least 30% in an ME drive, the RI drives suck at consistent performance). If those can be reduced by more vertical integration that would reduce costs of the systems and make for better overall performance as well.
- jobu 11y ago
  It amazes me that SSD drives still (mostly) conform to the same drive sizes as spinning disks. My most recent computer build used an M.2 slot SSD, and plugging that tiny little card into the motherboard seemed like the future. No cables required, and it was even smaller than the RAM sticks.
pbhjpbhj 11y ago
I'd imagine it's more likely to be the engineering tolerances required to keep the vertical movement to a minimum at double the radius? But I have no real insight, just a guess. It does strike me though - why do disks do serial reads, couldn't the head read several tracks at once? Why not have several heads angularly spaced around the drive? Presumably such things have been tried and found lacking in some respect. Looks like I need to do some research ...
- eru 11y ago
  There were some disks with two heads once. They were mechanically too complicated.
- avian 11y ago
  > couldn't the head read several tracks at once? The paper in fact suggests this and also cites an attempt by Conner to introduce multiple actuator drives: http://www.pcguide.com/ref/hdd/op/actMultiple-c.html http://www.pcguide.com/ref/hdd/op/actMultiple-c.html
noahdesu 11y ago
On Tuesday Eric Brewer gave a talk on this white paper at FAST. In the talk (and probably discussed more in the paper) he proposed looking at alternative disk form factors. He seemed to be particularly interested in "tall" disks, say 2U, that had the same surface area as today, but with more platters, and potentially more heads.
- Sanddancer 11y ago
  Ah, going back to the beginning of micros, where drives were often full-height things that could easily break feet when dropped. The thing that worries me there is that having had a big full-height 9 gb seagate scsi drive back in the day is that you needed to give the drive about a hundred watts on its own to run it, with a spool up that sounded like a jet engine was about to take off. I know that motors have gotten faster since then, and platters have gotten lighter, but that's still quite a bit of mass to get up to speed. The thing that interests me more is the idea of more read/write heads. Add a second, independent set of heads for read/write and you will decrease latency a good amount. Or, a second dependent set of heads which would not provide as much of a performance boost, but would still improve latency a good amount.
jabl 11y ago
> Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? > Why not go back to 8 inch for massive surface area? Or is that too much mass to spin. One problem AFAIK is that larger disks wobble too much, more the faster they spin. Which is why if you break open a 3.5" 15k drive you'll see that the platters are more like 2.5", and a lot of wasted space. For 7.2k you can use bigger platters, but how much bigger? I doubt 8" is feasible. Of course you can reduce the rpm further, but that has downsides too.
gshubert17 11y ago
Current disk densities are about 10^12 bits/square inch. To get 800 terabits on a single platter, it would need a diameter of about 32 inches. At 10^14 bits/square inch, only 3.2 inches. Assuming a compound annual growth rate of 25%, densities will be 10-fold larger in 10 years, and 100-fold larger in 20 years. (This actually happened in hard disks between 1972 and 1992.) But that would take us to 2035, not 2020. At those densities, a bit will have to be stored inside a square about 7 or 8 nanometers on a side. I don't know whether that's possible.

baruch 11y ago

A big thing they are saying in the paper is that they want to see data center HDDs that care slightly less about their medium error rates and opt to give better areal density and more consistent tail latencies and let the system above it to handle the reliability as it needs to do anyway. I've been thinking like this for a while and would tend to reduce the ERC (Error Recovery Control) to a minimum but the disks are still not designed to have these at very low numbers and Google has several interesting ideas in their paper along these lines. It would be really awesome if they get the HDD makers to go along with this.

davidiach 11y ago

"For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage every day or about 100x the Library of Congress. As shown in the graph, this continues to grow exponentially, with a 10x increase every five years." This blew my mind. What does this even mean in terms of logistics. How many people do you need to have just to add all those hard drives? How many new datacenters do you need to build every 5 years?

nostrademons 11y ago
It's actually not that much. Current consumer desktop hard drives top out at 8T (you can get enterprise RAID boxes of up to 48T now, but leave them out of the equation). 1P/day ~= 128 hard drives, so at 4min/drive, that's one person. Of course, that's just YouTube, and Google has many other needs for data. But people forget just how big the denominators are on this quantity, and how effective Kryder's Law has been. They also forget how much reserve capacity there is in human labor; Google's datacenters have tiny employee counts because they are so automated, and could easily scale up into the exabyte/day range. A more interesting question is what the differential rates of Kryder's Law vs. Moore's Law will do to how we architect software. Already, people in the know say that "disk is the new tape" - disk drive capacity has been increasing much faster than seek times, bus bandwidth, and available processing power, which means that you have to start treating the drive as a sequential storage device and not as a random-access platter. That's behind a lot of the shift from B-trees (as in conventional RDBMSes) to LSM-trees (as in BigTable/LevelDB), and also the resurgence of batch-processing frameworks like MapReduce. How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive?
- msellout 11y ago
  > How does the software you build change when reading & writing data sequentially is really cheap, but accessing it randomly is expensive? I thought we were already in that situation. Cache is king.
  - 11y ago

deleted 11y ago

[deleted]

deleted 11y ago
[deleted]

deleted 11y ago

[deleted]

ibmthrowaway271 11y ago

> For example, for YouTube alone, users upload over 400 hours of video every minute, which at one gigabyte per hour requires more than one petabyte (1M GB) of new storage every day or about 100x the Library of Congress Hmm, something up with the sums in the middle of that. 400 hours of video every minute is much more than one gigabyte per hour. It's way more than one terabyte per hour. Working backwards:- 1 PB/day =~ 42 TB/hour =~ 728 GB/minute

lcpriest 11y ago
I believe they are saying that the 400 hours they upload every minute are 400GB worth of data.
- ibmthrowaway271 11y ago
  Ah, yes, good point.
fluxquanta 11y ago
Variable compression and video resolution?
ucha 11y ago
400 hours video/min = 400 GB/min = 400 x 60 x 24 GB/day = 0.576 PB/day. Close enough.
jfasi 11y ago
That's only if you're storing one copy of the video.

novaleaf 11y ago

reminds me of the 5.25" Quantum Bigfoot. I'd be happy with a big disk like that in a storage network / raid. https://en.wikipedia.org/wiki/Quantum_Bigfoot https://en.wikipedia.org/wiki/Quantum_Bigfoot

mapmap 11y ago
The article doesn't go into much detail on the performance. I'm curious if reading was any faster because of the relatively faster spin speed at the outer rings of a larger disk? Or if it was a wash with seek times increasing due to the larger area?
dzhiurgis 11y ago
Which reminds me a teardown EEVblog did on 1980s 11" mainframe drive: https://www.youtube.com/watch?v=CBjoWMA5d84 https://www.youtube.com/watch?v=CBjoWMA5d84

cfcef 11y ago

The point about ECC and accepting lower error rates makes a lot of sense. It's the end-to-end principle again: since all of these HDDs are going into a global pool in which each of them is disposable and written content is protected by FEC spread across multiple drives, there is no need for each drive to spend a lot of resources, going deep into diminishing returns, trying to make itself as resilient as possible. If a network transmission fails, it is retried and doesn't need to be swathed in huge numbers of elaborate checksums and ultra-reliable links; if a hard drive fails, the content is recovered from the FEC and re-written out to a new drive.