6 ms·

Should be interesting to see what they come up with. Somehow I don't think they will find things better than multi-million dollar research like HAMR for write-

by ck2 11y ago

Should be interesting to see what they come up with. Somehow I don't think they will find things better than multi-million dollar research like HAMR for write-once read-many, but we'll see I guess. I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? Why not go back to 8 inch for massive surface area? Or is that too much mass to spin.

withjive 11y ago
Indeed, it will be interesting to see what Google can think up compared to the other big researchers. Though I wonder if the goals are slightly different, as Google is focusing on "Cloud Storage", rather than single hard performance. Which as they mentioned, the Cloud doesn't need to have great reliability (since data is assumed to be already redundant), this may allow them to decrease the cost of hard drives for exchange in cheaper disks that may fail more. Doesn't sound like they are expecting to increase size/performance alone with this effort.
StreamBright 11y ago
In the long run anything that moves in DC is way less reliable than systems without moving parts. On the top of that, these systems also need more energy to maintain the movement. If you increase the size than you need better materials to keep up with the increased mechanical load. I bet by 2020 we are going to have better ways of storing large amount of data than a spinning disk. The new optical drive would be a good candidate for some use cases, mostly for archiving though: http://www.sciencealert.com/this-new-5d-data-storage-disc-can-store-360tb-of-data-for-14-billion-years http://www.sciencealert.com/this-new-5d-data-storage-disc-ca... For continuous read/write workloads SSDs are the best for many reasons (energy efficiency, latency, IO bandwidth) and I think the world is moving that direction to replace spinning disks with these. I am hoping to see a bump in the SSD capacity as well. We are here now: http://www.extremetech.com/computing/221303-the-worlds-biggest-ssd-has-arrived-at-13tb http://www.extremetech.com/computing/221303-the-worlds-bigge...
- ucaetano 11y ago
  From the article: "An obvious question is why are we talking about spinning disks at all, rather than SSDs, which have higher IOPS and are the “future” of storage." "The root reason is that the cost per GB remains too high, and more importantly that the growth rates in capacity/$ between disks and SSDs are relatively close (at least for SSDs that have sufficient numbers of program-erase cycles to use in data centers), so that cost will not change enough in the coming decade."
  - zaroth 11y ago
    Well at least for YouTube video storage, I imagine the P/E cycles are not a problem. There have been issues with longevity like on the Samsung 840 EVO drives, but if you can avoid that then you can build some incredibly dense, fast, reliable and power efficient storage on top of V-NAND. Cost per GB for SSDs is definitely catching up to HDD [1] and I think is expected to match or beat HDD within the decade. Power efficiency should be better on SSD and MTBF will be much better than HDD, so I would have thought TCO for SSD would have HDDs basically beat already. [1] https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-hdd-chart-retry-100631270-orig.png https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-...
cm2187 11y ago
For 100TB disks to make sense the read/write speed would have to improve considerably, otherwise the disk will be useless other than for cold storage. I haven't really seen spinning disks speed improving much. It would take weeks to rebuild an array of 100TB disks at the current speed.
- rbanffy 11y ago
  I remember a washing machine sized IBM mainframe hard disk with tho sets of heads. That's one trivial way to double bandwidth and cut latency (it'll read like a RAID-1 and write as a RAID-0). Also, the more platters, the more data you can read on a single rotation.
  - cm2187 11y ago
    You double bandwidth for a single access. But if you are storing 100TB on a single drive, it's unlikely to be 100TB related to a single user. You will want to make a lot of requests to this data simultaneously, a lot more requests than what you would to a disk containing only 1TB of data.
- reitzensteinm 11y ago
  100TB should only be 5x (i.e. sqrt( 25 )) slower to access each byte sequentially than a 4TB drive (although shingling will screw up this calculation, since it provides more tracks, but not more density on each track). Which is a long time, but not totally out of bounds for the way we're currently using disks. The coming decade is going to be very interesting, SSDs are pushing forward on an insane trajectory, which could either fall apart as the benefit of node shrinks disappear, or accelerate through a virtuous cycle as unit volume increases. It's even plausible that spinning rust is on death's door in that time frame. The price floor of SSDs is much lower than that of hard drives (i.e. how much does an 8gb hard drive cost to manufacture today in volume? As much as a 500 gb drive), meaning that both mildly performance conscious and price sensitive consumers both move entirely over. Enterprise customers will still chase the price/gb and storage density of traditional hard drives, but without the consumer dollars to invest in R&D, that could start to stall, quickly leaving them with diminishing advantage and obsolete in the blink of an eye.
darkr 11y ago
> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise). Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure
- SixSigma 11y ago
  > you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.
- __david__ 11y ago
  > Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.
  - ck2 11y ago
    Is that defaulting to EXT4 or XFS ?
simplemath 11y ago
If they go with anything Arm and Platter I'd be shocked. I'm imagining a 10 acre array of NAND
- baruch 11y ago
  They are saying the paper they want arms and platters as it's more economical. An HDD is cheaper than an SSD of the same capacity at four to ten times and I don't think the electricity cost will offset this if you care more about capacity than performance.
  - rbanffy 11y ago
    It is right now, but silicon and precision mechanics scale with different curves. There is a point where spinning metal will no longer be viable as a competitor and I'll guess, based on a hunch, it's not that far into the future.
    - tytso 11y ago
      Flash is not getting cheaper faster than HDD's, once you control for write endurance. One of the big messages in the paper is that IOPS matter. If you only have a petabyte or two of data, you can use a brute force solution such as flash or 15k drives. But at Cloud scale (we used to say Google scale, but there other companies like Amazon and Microsoft and Facebook that run at these scales now) you just can't afford that much flash. So the trick is to use every last IOPS that you can out of each and every disk, and that means provisioning and spreading out your data across an entire fleet of disks. The alternative, where you silo your storage in disk pools, and where some disks or disk pools have IOPS that are unused --- wasted, and where you make up for that by using flash for your hot and warm workloads is just a much more expensive way to do things. It's for that reason that SMR drives aren't all that interesting. Sure, you get 20% more capacity, but at the cost of burning most of your IOPS for GC. Or if you move all of your SMR-friendly data to the SMR drives, that's cold data, and it means that you've wasted all of the IOPS that those SMR drives are capable of. Which means the rest of your disk fleet is now much hotter, and you might have to buy more CMR spindles to cover the cost --- at which point the cost advantage of SMR drives disappears. (I helped to author the section about SMR drives, and why we are interested in hybrid SMR/CMR drives as a solution to this problem; read the paper for more details.)
Cthulhu_ 11y ago
I for one am confident that SSD and its technology will be much more scalable and future-proof than spinning disks; stackable chips are a thing already. I'm sure that by 2020, there will be companies that can deliver a storage platform with petabytes of space in a 1U package. Just imagine one of those server things crammed full of ssd chips.
- baruch 11y ago
  In the long run that may be true, but the same things they talk about (minus the mechanical head changes) will still hold in SSDs. The background activity is even worse for SSDs and the tail latency still very much exists. Essentially the problem is that the HDDs and SSDs are built as black boxes and are not vertically integrated with a data center storage scheme. The major win will be when the HDDs and SSDs are slightly less self-sufficient and let the higher components in the stack manage them more in depth. Being able to tell the disk "do not bother with ecc for now" and try to get the data from your RAID and if that doesnt work come back to the disk and tell it "we really really want the data, spend as much as needed on this" is very strong in both getting the tail latency down and getting the reliability to stay at the same level. For SSDs there is an added complication of garbage collection and the incredible amounts of over-provisioning needed to get consistent performance from them (at least 30% in an ME drive, the RI drives suck at consistent performance). If those can be reduced by more vertical integration that would reduce costs of the systems and make for better overall performance as well.
- jobu 11y ago
  It amazes me that SSD drives still (mostly) conform to the same drive sizes as spinning disks. My most recent computer build used an M.2 slot SSD, and plugging that tiny little card into the motherboard seemed like the future. No cables required, and it was even smaller than the RAM sticks.
pbhjpbhj 11y ago
I'd imagine it's more likely to be the engineering tolerances required to keep the vertical movement to a minimum at double the radius? But I have no real insight, just a guess. It does strike me though - why do disks do serial reads, couldn't the head read several tracks at once? Why not have several heads angularly spaced around the drive? Presumably such things have been tried and found lacking in some respect. Looks like I need to do some research ...
- eru 11y ago
  There were some disks with two heads once. They were mechanically too complicated.
- avian 11y ago
  > couldn't the head read several tracks at once? The paper in fact suggests this and also cites an attempt by Conner to introduce multiple actuator drives: http://www.pcguide.com/ref/hdd/op/actMultiple-c.html http://www.pcguide.com/ref/hdd/op/actMultiple-c.html
noahdesu 11y ago
On Tuesday Eric Brewer gave a talk on this white paper at FAST. In the talk (and probably discussed more in the paper) he proposed looking at alternative disk form factors. He seemed to be particularly interested in "tall" disks, say 2U, that had the same surface area as today, but with more platters, and potentially more heads.
- Sanddancer 11y ago
  Ah, going back to the beginning of micros, where drives were often full-height things that could easily break feet when dropped. The thing that worries me there is that having had a big full-height 9 gb seagate scsi drive back in the day is that you needed to give the drive about a hundred watts on its own to run it, with a spool up that sounded like a jet engine was about to take off. I know that motors have gotten faster since then, and platters have gotten lighter, but that's still quite a bit of mass to get up to speed. The thing that interests me more is the idea of more read/write heads. Add a second, independent set of heads for read/write and you will decrease latency a good amount. Or, a second dependent set of heads which would not provide as much of a performance boost, but would still improve latency a good amount.
jabl 11y ago
> Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? > Why not go back to 8 inch for massive surface area? Or is that too much mass to spin. One problem AFAIK is that larger disks wobble too much, more the faster they spin. Which is why if you break open a 3.5" 15k drive you'll see that the platters are more like 2.5", and a lot of wasted space. For 7.2k you can use bigger platters, but how much bigger? I doubt 8" is feasible. Of course you can reduce the rpm further, but that has downsides too.
gshubert17 11y ago
Current disk densities are about 10^12 bits/square inch. To get 800 terabits on a single platter, it would need a diameter of about 32 inches. At 10^14 bits/square inch, only 3.2 inches. Assuming a compound annual growth rate of 25%, densities will be 10-fold larger in 10 years, and 100-fold larger in 20 years. (This actually happened in hard disks between 1972 and 1992.) But that would take us to 2035, not 2020. At those densities, a bit will have to be stored inside a square about 7 or 8 nanometers on a side. I don't know whether that's possible.

by ck2 11y ago

withjive 11y ago

Indeed, it will be interesting to see what Google can think up compared to the other big researchers. Though I wonder if the goals are slightly different, as Google is focusing on "Cloud Storage", rather than single hard performance. Which as they mentioned, the Cloud doesn't need to have great reliability (since data is assumed to be already redundant), this may allow them to decrease the cost of hard drives for exchange in cheaper disks that may fail more. Doesn't sound like they are expecting to increase size/performance alone with this effort.

StreamBright 11y ago

In the long run anything that moves in DC is way less reliable than systems without moving parts. On the top of that, these systems also need more energy to maintain the movement. If you increase the size than you need better materials to keep up with the increased mechanical load. I bet by 2020 we are going to have better ways of storing large amount of data than a spinning disk. The new optical drive would be a good candidate for some use cases, mostly for archiving though: http://www.sciencealert.com/this-new-5d-data-storage-disc-can-store-360tb-of-data-for-14-billion-years http://www.sciencealert.com/this-new-5d-data-storage-disc-ca... For continuous read/write workloads SSDs are the best for many reasons (energy efficiency, latency, IO bandwidth) and I think the world is moving that direction to replace spinning disks with these. I am hoping to see a bump in the SSD capacity as well. We are here now: http://www.extremetech.com/computing/221303-the-worlds-biggest-ssd-has-arrived-at-13tb http://www.extremetech.com/computing/221303-the-worlds-bigge...

ucaetano 11y ago
From the article: "An obvious question is why are we talking about spinning disks at all, rather than SSDs, which have higher IOPS and are the “future” of storage." "The root reason is that the cost per GB remains too high, and more importantly that the growth rates in capacity/$ between disks and SSDs are relatively close (at least for SSDs that have sufficient numbers of program-erase cycles to use in data centers), so that cost will not change enough in the coming decade."
- zaroth 11y ago
  Well at least for YouTube video storage, I imagine the P/E cycles are not a problem. There have been issues with longevity like on the Samsung 840 EVO drives, but if you can avoid that then you can build some incredibly dense, fast, reliable and power efficient storage on top of V-NAND. Cost per GB for SSDs is definitely catching up to HDD [1] and I think is expected to match or beat HDD within the decade. Power efficiency should be better on SSD and MTBF will be much better than HDD, so I would have thought TCO for SSD would have HDDs basically beat already. [1] https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-hdd-chart-retry-100631270-orig.png https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-...

cm2187 11y ago

For 100TB disks to make sense the read/write speed would have to improve considerably, otherwise the disk will be useless other than for cold storage. I haven't really seen spinning disks speed improving much. It would take weeks to rebuild an array of 100TB disks at the current speed.

rbanffy 11y ago
I remember a washing machine sized IBM mainframe hard disk with tho sets of heads. That's one trivial way to double bandwidth and cut latency (it'll read like a RAID-1 and write as a RAID-0). Also, the more platters, the more data you can read on a single rotation.
- cm2187 11y ago
  You double bandwidth for a single access. But if you are storing 100TB on a single drive, it's unlikely to be 100TB related to a single user. You will want to make a lot of requests to this data simultaneously, a lot more requests than what you would to a disk containing only 1TB of data.
reitzensteinm 11y ago
100TB should only be 5x (i.e. sqrt( 25 )) slower to access each byte sequentially than a 4TB drive (although shingling will screw up this calculation, since it provides more tracks, but not more density on each track). Which is a long time, but not totally out of bounds for the way we're currently using disks. The coming decade is going to be very interesting, SSDs are pushing forward on an insane trajectory, which could either fall apart as the benefit of node shrinks disappear, or accelerate through a virtuous cycle as unit volume increases. It's even plausible that spinning rust is on death's door in that time frame. The price floor of SSDs is much lower than that of hard drives (i.e. how much does an 8gb hard drive cost to manufacture today in volume? As much as a 500 gb drive), meaning that both mildly performance conscious and price sensitive consumers both move entirely over. Enterprise customers will still chase the price/gb and storage density of traditional hard drives, but without the consumer dollars to invest in R&D, that could start to stall, quickly leaving them with diminishing advantage and obsolete in the blink of an eye.

darkr 11y ago

> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise). Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure

SixSigma 11y ago
> you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.
__david__ 11y ago
> Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.
- ck2 11y ago
  Is that defaulting to EXT4 or XFS ?

simplemath 11y ago

If they go with anything Arm and Platter I'd be shocked. I'm imagining a 10 acre array of NAND

baruch 11y ago
They are saying the paper they want arms and platters as it's more economical. An HDD is cheaper than an SSD of the same capacity at four to ten times and I don't think the electricity cost will offset this if you care more about capacity than performance.
- rbanffy 11y ago
  It is right now, but silicon and precision mechanics scale with different curves. There is a point where spinning metal will no longer be viable as a competitor and I'll guess, based on a hunch, it's not that far into the future.
  - tytso 11y ago
    Flash is not getting cheaper faster than HDD's, once you control for write endurance. One of the big messages in the paper is that IOPS matter. If you only have a petabyte or two of data, you can use a brute force solution such as flash or 15k drives. But at Cloud scale (we used to say Google scale, but there other companies like Amazon and Microsoft and Facebook that run at these scales now) you just can't afford that much flash. So the trick is to use every last IOPS that you can out of each and every disk, and that means provisioning and spreading out your data across an entire fleet of disks. The alternative, where you silo your storage in disk pools, and where some disks or disk pools have IOPS that are unused --- wasted, and where you make up for that by using flash for your hot and warm workloads is just a much more expensive way to do things. It's for that reason that SMR drives aren't all that interesting. Sure, you get 20% more capacity, but at the cost of burning most of your IOPS for GC. Or if you move all of your SMR-friendly data to the SMR drives, that's cold data, and it means that you've wasted all of the IOPS that those SMR drives are capable of. Which means the rest of your disk fleet is now much hotter, and you might have to buy more CMR spindles to cover the cost --- at which point the cost advantage of SMR drives disappears. (I helped to author the section about SMR drives, and why we are interested in hybrid SMR/CMR drives as a solution to this problem; read the paper for more details.)

Cthulhu_ 11y ago

I for one am confident that SSD and its technology will be much more scalable and future-proof than spinning disks; stackable chips are a thing already. I'm sure that by 2020, there will be companies that can deliver a storage platform with petabytes of space in a 1U package. Just imagine one of those server things crammed full of ssd chips.

baruch 11y ago
In the long run that may be true, but the same things they talk about (minus the mechanical head changes) will still hold in SSDs. The background activity is even worse for SSDs and the tail latency still very much exists. Essentially the problem is that the HDDs and SSDs are built as black boxes and are not vertically integrated with a data center storage scheme. The major win will be when the HDDs and SSDs are slightly less self-sufficient and let the higher components in the stack manage them more in depth. Being able to tell the disk "do not bother with ecc for now" and try to get the data from your RAID and if that doesnt work come back to the disk and tell it "we really really want the data, spend as much as needed on this" is very strong in both getting the tail latency down and getting the reliability to stay at the same level. For SSDs there is an added complication of garbage collection and the incredible amounts of over-provisioning needed to get consistent performance from them (at least 30% in an ME drive, the RI drives suck at consistent performance). If those can be reduced by more vertical integration that would reduce costs of the systems and make for better overall performance as well.
jobu 11y ago
It amazes me that SSD drives still (mostly) conform to the same drive sizes as spinning disks. My most recent computer build used an M.2 slot SSD, and plugging that tiny little card into the motherboard seemed like the future. No cables required, and it was even smaller than the RAM sticks.

pbhjpbhj 11y ago

I'd imagine it's more likely to be the engineering tolerances required to keep the vertical movement to a minimum at double the radius? But I have no real insight, just a guess. It does strike me though - why do disks do serial reads, couldn't the head read several tracks at once? Why not have several heads angularly spaced around the drive? Presumably such things have been tried and found lacking in some respect. Looks like I need to do some research ...

eru 11y ago
There were some disks with two heads once. They were mechanically too complicated.
avian 11y ago
> couldn't the head read several tracks at once? The paper in fact suggests this and also cites an attempt by Conner to introduce multiple actuator drives: http://www.pcguide.com/ref/hdd/op/actMultiple-c.html http://www.pcguide.com/ref/hdd/op/actMultiple-c.html

noahdesu 11y ago

On Tuesday Eric Brewer gave a talk on this white paper at FAST. In the talk (and probably discussed more in the paper) he proposed looking at alternative disk form factors. He seemed to be particularly interested in "tall" disks, say 2U, that had the same surface area as today, but with more platters, and potentially more heads.

Sanddancer 11y ago
Ah, going back to the beginning of micros, where drives were often full-height things that could easily break feet when dropped. The thing that worries me there is that having had a big full-height 9 gb seagate scsi drive back in the day is that you needed to give the drive about a hundred watts on its own to run it, with a spool up that sounded like a jet engine was about to take off. I know that motors have gotten faster since then, and platters have gotten lighter, but that's still quite a bit of mass to get up to speed. The thing that interests me more is the idea of more read/write heads. Add a second, independent set of heads for read/write and you will decrease latency a good amount. Or, a second dependent set of heads which would not provide as much of a performance boost, but would still improve latency a good amount.

jabl 11y ago

> Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter? > Why not go back to 8 inch for massive surface area? Or is that too much mass to spin. One problem AFAIK is that larger disks wobble too much, more the faster they spin. Which is why if you break open a 3.5" 15k drive you'll see that the platters are more like 2.5", and a lot of wasted space. For 7.2k you can use bigger platters, but how much bigger? I doubt 8" is feasible. Of course you can reduce the rpm further, but that has downsides too.

gshubert17 11y ago

Current disk densities are about 10^12 bits/square inch. To get 800 terabits on a single platter, it would need a diameter of about 32 inches. At 10^14 bits/square inch, only 3.2 inches. Assuming a compound annual growth rate of 25%, densities will be 10-fold larger in 10 years, and 100-fold larger in 20 years. (This actually happened in hard disks between 1972 and 1992.) But that would take us to 2035, not 2020. At those densities, a bit will have to be stored inside a square about 7 or 8 nanometers on a side. I don't know whether that's possible.