7 ms·

As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while perf

As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while performance remains darn near constant, so the gap between the two keeps getting wider. Yes, we can use flash to absorb the I/O demand, treat disk more like tape, yadda yadda, but all of that takes significant effort. You do want your storage system to be correct despite the greater complexity of heterogeneous hardware, don't you? Looking on the bright side: infinite job security. Yay?

bunnycorn 8y ago
I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is generally not requested.
- notacoward 8y ago
  > a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too. In my world, the professionals very much do care about latency. Yes, the systems are built for massive throughput, but that throughput has to be within a certain latency. If latency goes too high, we get calls in the middle of the night. To keep that system-wide latency low, we deploy many racks' worth of equipment at a time, or shift load between similarly sized parts of the system. For us and our peers, this capacity/performance gap is a huge issue. Has been for years, and it keeps getting worse. Those concerns might not apply to everyone, notably you, but they are quite real.
  - bunnycorn 8y ago
    And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.
    - notacoward 8y ago
      Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectations even with hard disks. Those requirements might not be as stringent for hard disks as for flash (or for distributed vs. local) but they're very much still there. If a significant fraction of users' requests are taking too long, they do complain. Loudly. Data scientists don't care if the system is simultaneously delivering dozens of gigabytes per second to other users, and doing ten kinds of background maintenance stuff besides. They care that their job is slow. When you're dealing with truly large amounts of data being processed by thousands of machines, and ad hoc queries none of which are likely to be repeated, "just use flash" isn't an answer. Maybe you're dealing with a different kind of professional than I am. That's fine, but you shouldn't keep making these super-general statements that are wrong for one of the largest classes of data professionals. "Couple of hard drives" was so far off the mark I literally laughed out loud.
      - bunnycorn 8y ago
        No, I didn't. It was your problem you misinterpreted what I said. Nobody else misunderstood me but you.
        notacoward 8y ago
        > Nobody else misunderstood me but you. I think it would be more correct to say that nobody at all misunderstood you, and nobody else bothered to correct your claims. I do see there were some downvotes, though, which suggests what others thought of them.
        bunnycorn 8y ago
        There was one downvote: yours.
zzzcpan 8y ago
But it was always like that for HDDs. You always had to assume no change in seek time with more capacity. And it was always wrong to treat HDDs as random IO devices with all the random writes and b-trees. At least now there is a pressure on everyone involved to do it right.
- notacoward 8y ago
  That's a completely different issue. Yes, it's important to do disk accesses right. That was even more true when I started doing this stuff in 1990 than it is now, because disks back then didn't have big caches or fancy controllers, and RAID wasn't a thing. My point is that even if you do everything right it's getting harder and harder for a disk-based system to keep up with I/O demand. That means more and more gyrations to keep the demand away from those disk-based systems, which means more overall system complexity. In particular, more places for inconsistency to creep in. As though we didn't have enough to deal with making these things distributed.
  - zzzcpan 8y ago
    I get it, but take RAID for example. It always presumed weird things. That you either have enough performance and not enough load to be able to check disks or rebuild it without much degradation or that you are able to copy an entire disk in like a few hours or that an error means you need to replace the disk. These assumptions break if you don't forget that capacity increases, but seek time stays the same. Obviously there will be some inherent complexity dealing with that, but not really more complexity, it's just different, maybe proper for once.
    - notacoward 8y ago
      I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next one was scheduled. Ah, the good old days. That was the easy case. The system I work on today has a dozen such maintenance activities going on all the time. Making sure they all complete in reasonable time, without affecting the endless stream of new user I/O too much, requires increasingly sophisticated scheduling as the gap gets wider. That's a lot of complexity right there. On top of that, there's this little formula: load = warmth * capacity Warmth is effectively constant for relevant time scales, so increasing capacity means increasing load. Scaling out doesn't help in this case, not even with perfect linear scaling, because every bit of added capability brings added capacity and load with it. Until you run out of users, I guess, but those seem to be infinite. ;) Thus, systems with the same basic architecture get more and more overloaded as per-disk capacity increases, until they become useless. The only escape is to change the architecture, inevitably toward greater complexity (e.g. cache layers and burst buffers). None of this would be necessary with a better capacity/performance ratio. Believe me, nobody wants to make these systems more complicated. But every time that gap gets bigger, there will likely be a new increment of complexity to deal with it I don't see how that's "proper" or "where it belongs" or any such. It's not that all such systems were poorly designed for the hardware as it was when they were developed. It's that even the best designs have to keep adapting. The demise of Moore's law and the ever increasing number of cores per die or per system have increased complexity in the compute domain. The capacity/performance gap is the storage equivalent.
blattimwind 8y ago
> Capacity keeps increasing while performance remains darn near constant, The recent capacity increases seem to be mostly driven by stuffing more platters in, so as long as they don't have independently servoed heads (used to be a thing, maybe it becomes a thing again? Perhaps not fully independent, but still "ganged up" with some sort of micromechanics in each head to servo it to nearby tracks?) or increase data density (bits/cm^2), throughput must plateau.
kalleboo 8y ago
Even as a home user, expanding a RAID on a consumer NAS with a 10 TB harddisk takes days due to the size. It would take weeks with a 100 TB drive! (then again, that's not something you'd need to do often with 100 TB to play with :)
- StillBored 8y ago
  The bandwidth of these drives are increasing due to the fact that the rotational speeds are tending to remain the same while the density is increasing. So, as the linear in track BPI increases so does the throughput. Tracks per inch, which is where most of the density has been doesn't improve bandwidth. This is where a lot of the density improvements a few years ago was coming from (higher TPI). At one point the TPI was actually higher than the linear BPI, which was something completely non-obvious to me when I discovered it. HAMR is an increase in linear BPI, so expect these drives to get faster.
Aloha 8y ago
The part that makes me recoil in horror is the potential damage wrought by one failure - at what point does the rebuild time for your RAID Array start to collide with the MTBF for the drives? ;-)
- calcifer 8y ago
  We are getting there, this already happens with unrecoverable read errors (URE). With RAID5 for example, rebuilding a 4TiB array is expected (i.e. >50% chance) to have at least one URE.
- notacoward 8y ago
  This is why the really big storage systems use erasure coding. At scale, RAID-5 and even RAID-6 are vulnerable to these kinds of overlapping failures. Not theory; seen it happen.
StillBored 8y ago
OTOH, I was in the storage industry until recently, and HD capacity increases were always welcome. That is because we were a bandwidth play, sold on the basis of density and capacity. Hello backup and archive. That is frankly where most of HD storage is today, 40 drives sitting in 4U is still a few ten-thousand ops per second, but it has a few tends of GB/sec of bandwidth too. So, if your access patterns are long linear read/writes then HDD's work just fine. Even for midrange IOP applications, if you burst a couple GB from disk to a SSD cache, its possible to absorb a lot of IOP's. Also, rebuild/scrub operations aren't really an issue either if your running RAID6, and your system is sufficiently designed/over provisioned to deal with rebuild/scrub operations happening in the background. If your IOP limited OTOH, you likely have a problem.
- notacoward 8y ago
  > Hello backup and archive. That is frankly where most of HD storage is today I'm not sure whether that's true or not. At one end, tape still exists. At the other end, a lot of data that's already stored with high levels of redundancy doesn't get backed up anywhere else. Do you have a good source for a definitive answer? > rebuild/scrub operations aren't really an issue either if your running RAID6 Rebuild/scrub operations still happen, and are still an issue, with RAID6. They might not be quite as visible because they're not soaking up host cycles, but they are soaking up disk IOPS. In any large or even medium-sized storage infrastructure (by today's standards) you'll have some going on and occasionally you'll have some that overlap because the first one took too long. Before long you'll hit a case where you get a third failure while the first two are still going on, and you stop relying on insufficient RAID6. Then you're into erasure codes and your own kind of scrubbing. Those aren't exotic situations or responses any more. > If your IOP limited OTOH, you likely have a problem. Yep, sure do. Can't wish it away, or make it all better with a magical free SSD caching layer. Have to solve it, which requires effort and expense. What I'm saying is that bigger disks make that harder. I'm not complaining, it's what I and others choose to do, but it's a fact.
  - StillBored 8y ago
    I don't have a direct summary link, but if you read random storage revenue/capacity reports that seems to be the case. There are a few large buyers of bulk storage which appear to be using it for some form of nearline. At least in my experience (which is obviously warped by the part of the industry I was in), enterprise applications needing IOP's have been overwhelmingly moving to pure flash. There remains a large amount of revenue in hybrid arrays, but the volume is shrinking (although maybe not the raw capacity, similar to the mainframe which is selling record amounts of capacity in fewer and fewer machines). A few years ago I stopped being surprised to see a couple racks of infortrend's or supermicro storage chassis (or any number of other 2nd tier vendors products) sitting at one end of random datacenters, where the local storage admins were running some huge snapshot repo, or Ceph, or whatever on them and the resulting capacities were frequently powers of ten greater than the online storage. Also, I've seen plenty of tape arrays too, but they don't seem nearly as common these days. Partially, because tape has the same problems as disk (lots of bandwidth, but it never seems to be enough). It seems everyone still has one, but you have to hunt for it, and it might only be getting the most critical data (or stuff that is required for compliance with some law), which turns out to be limited by the 2-4-8 drives they have constantly spinning, where an operator walks in every few days and swaps a couple dozen tapes offsite. Sure there are larger libraries but it seems most admin's start any conversation about tape with a groan and eye rolling, which tends to be an attitude that keeps finite resources from being heavily invested in them. So, people will buy 10+Gbit links between their data centers (which sometimes tend to be shockingly inexpensive) rather than spend $10k on a tape library and a fedex account.

bunnycorn 8y ago

I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is generally not requested.

notacoward 8y ago
> a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too. In my world, the professionals very much do care about latency. Yes, the systems are built for massive throughput, but that throughput has to be within a certain latency. If latency goes too high, we get calls in the middle of the night. To keep that system-wide latency low, we deploy many racks' worth of equipment at a time, or shift load between similarly sized parts of the system. For us and our peers, this capacity/performance gap is a huge issue. Has been for years, and it keeps getting worse. Those concerns might not apply to everyone, notably you, but they are quite real.
- bunnycorn 8y ago
  And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.
  - notacoward 8y ago
    Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectations even with hard disks. Those requirements might not be as stringent for hard disks as for flash (or for distributed vs. local) but they're very much still there. If a significant fraction of users' requests are taking too long, they do complain. Loudly. Data scientists don't care if the system is simultaneously delivering dozens of gigabytes per second to other users, and doing ten kinds of background maintenance stuff besides. They care that their job is slow. When you're dealing with truly large amounts of data being processed by thousands of machines, and ad hoc queries none of which are likely to be repeated, "just use flash" isn't an answer. Maybe you're dealing with a different kind of professional than I am. That's fine, but you shouldn't keep making these super-general statements that are wrong for one of the largest classes of data professionals. "Couple of hard drives" was so far off the mark I literally laughed out loud.
    - bunnycorn 8y ago
      No, I didn't. It was your problem you misinterpreted what I said. Nobody else misunderstood me but you.
      - notacoward 8y ago
        > Nobody else misunderstood me but you. I think it would be more correct to say that nobody at all misunderstood you, and nobody else bothered to correct your claims. I do see there were some downvotes, though, which suggests what others thought of them.
        bunnycorn 8y ago
        There was one downvote: yours.

zzzcpan 8y ago

But it was always like that for HDDs. You always had to assume no change in seek time with more capacity. And it was always wrong to treat HDDs as random IO devices with all the random writes and b-trees. At least now there is a pressure on everyone involved to do it right.

notacoward 8y ago
That's a completely different issue. Yes, it's important to do disk accesses right. That was even more true when I started doing this stuff in 1990 than it is now, because disks back then didn't have big caches or fancy controllers, and RAID wasn't a thing. My point is that even if you do everything right it's getting harder and harder for a disk-based system to keep up with I/O demand. That means more and more gyrations to keep the demand away from those disk-based systems, which means more overall system complexity. In particular, more places for inconsistency to creep in. As though we didn't have enough to deal with making these things distributed.
- zzzcpan 8y ago
  I get it, but take RAID for example. It always presumed weird things. That you either have enough performance and not enough load to be able to check disks or rebuild it without much degradation or that you are able to copy an entire disk in like a few hours or that an error means you need to replace the disk. These assumptions break if you don't forget that capacity increases, but seek time stays the same. Obviously there will be some inherent complexity dealing with that, but not really more complexity, it's just different, maybe proper for once.
  - notacoward 8y ago
    I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next one was scheduled. Ah, the good old days. That was the easy case. The system I work on today has a dozen such maintenance activities going on all the time. Making sure they all complete in reasonable time, without affecting the endless stream of new user I/O too much, requires increasingly sophisticated scheduling as the gap gets wider. That's a lot of complexity right there. On top of that, there's this little formula: load = warmth * capacity Warmth is effectively constant for relevant time scales, so increasing capacity means increasing load. Scaling out doesn't help in this case, not even with perfect linear scaling, because every bit of added capability brings added capacity and load with it. Until you run out of users, I guess, but those seem to be infinite. ;) Thus, systems with the same basic architecture get more and more overloaded as per-disk capacity increases, until they become useless. The only escape is to change the architecture, inevitably toward greater complexity (e.g. cache layers and burst buffers). None of this would be necessary with a better capacity/performance ratio. Believe me, nobody wants to make these systems more complicated. But every time that gap gets bigger, there will likely be a new increment of complexity to deal with it I don't see how that's "proper" or "where it belongs" or any such. It's not that all such systems were poorly designed for the hardware as it was when they were developed. It's that even the best designs have to keep adapting. The demise of Moore's law and the ever increasing number of cores per die or per system have increased complexity in the compute domain. The capacity/performance gap is the storage equivalent.

blattimwind 8y ago

> Capacity keeps increasing while performance remains darn near constant, The recent capacity increases seem to be mostly driven by stuffing more platters in, so as long as they don't have independently servoed heads (used to be a thing, maybe it becomes a thing again? Perhaps not fully independent, but still "ganged up" with some sort of micromechanics in each head to servo it to nearby tracks?) or increase data density (bits/cm^2), throughput must plateau.

kalleboo 8y ago

Even as a home user, expanding a RAID on a consumer NAS with a 10 TB harddisk takes days due to the size. It would take weeks with a 100 TB drive! (then again, that's not something you'd need to do often with 100 TB to play with :)

StillBored 8y ago
The bandwidth of these drives are increasing due to the fact that the rotational speeds are tending to remain the same while the density is increasing. So, as the linear in track BPI increases so does the throughput. Tracks per inch, which is where most of the density has been doesn't improve bandwidth. This is where a lot of the density improvements a few years ago was coming from (higher TPI). At one point the TPI was actually higher than the linear BPI, which was something completely non-obvious to me when I discovered it. HAMR is an increase in linear BPI, so expect these drives to get faster.

Aloha 8y ago

The part that makes me recoil in horror is the potential damage wrought by one failure - at what point does the rebuild time for your RAID Array start to collide with the MTBF for the drives? ;-)

calcifer 8y ago
We are getting there, this already happens with unrecoverable read errors (URE). With RAID5 for example, rebuilding a 4TiB array is expected (i.e. >50% chance) to have at least one URE.
notacoward 8y ago
This is why the really big storage systems use erasure coding. At scale, RAID-5 and even RAID-6 are vulnerable to these kinds of overlapping failures. Not theory; seen it happen.

StillBored 8y ago

OTOH, I was in the storage industry until recently, and HD capacity increases were always welcome. That is because we were a bandwidth play, sold on the basis of density and capacity. Hello backup and archive. That is frankly where most of HD storage is today, 40 drives sitting in 4U is still a few ten-thousand ops per second, but it has a few tends of GB/sec of bandwidth too. So, if your access patterns are long linear read/writes then HDD's work just fine. Even for midrange IOP applications, if you burst a couple GB from disk to a SSD cache, its possible to absorb a lot of IOP's. Also, rebuild/scrub operations aren't really an issue either if your running RAID6, and your system is sufficiently designed/over provisioned to deal with rebuild/scrub operations happening in the background. If your IOP limited OTOH, you likely have a problem.

notacoward 8y ago
> Hello backup and archive. That is frankly where most of HD storage is today I'm not sure whether that's true or not. At one end, tape still exists. At the other end, a lot of data that's already stored with high levels of redundancy doesn't get backed up anywhere else. Do you have a good source for a definitive answer? > rebuild/scrub operations aren't really an issue either if your running RAID6 Rebuild/scrub operations still happen, and are still an issue, with RAID6. They might not be quite as visible because they're not soaking up host cycles, but they are soaking up disk IOPS. In any large or even medium-sized storage infrastructure (by today's standards) you'll have some going on and occasionally you'll have some that overlap because the first one took too long. Before long you'll hit a case where you get a third failure while the first two are still going on, and you stop relying on insufficient RAID6. Then you're into erasure codes and your own kind of scrubbing. Those aren't exotic situations or responses any more. > If your IOP limited OTOH, you likely have a problem. Yep, sure do. Can't wish it away, or make it all better with a magical free SSD caching layer. Have to solve it, which requires effort and expense. What I'm saying is that bigger disks make that harder. I'm not complaining, it's what I and others choose to do, but it's a fact.
- StillBored 8y ago
  I don't have a direct summary link, but if you read random storage revenue/capacity reports that seems to be the case. There are a few large buyers of bulk storage which appear to be using it for some form of nearline. At least in my experience (which is obviously warped by the part of the industry I was in), enterprise applications needing IOP's have been overwhelmingly moving to pure flash. There remains a large amount of revenue in hybrid arrays, but the volume is shrinking (although maybe not the raw capacity, similar to the mainframe which is selling record amounts of capacity in fewer and fewer machines). A few years ago I stopped being surprised to see a couple racks of infortrend's or supermicro storage chassis (or any number of other 2nd tier vendors products) sitting at one end of random datacenters, where the local storage admins were running some huge snapshot repo, or Ceph, or whatever on them and the resulting capacities were frequently powers of ten greater than the online storage. Also, I've seen plenty of tape arrays too, but they don't seem nearly as common these days. Partially, because tape has the same problems as disk (lots of bandwidth, but it never seems to be enough). It seems everyone still has one, but you have to hunt for it, and it might only be getting the most critical data (or stuff that is required for compliance with some law), which turns out to be limited by the 2-4-8 drives they have constantly spinning, where an operator walks in every few days and swaps a couple dozen tapes offsite. Sure there are larger libraries but it seems most admin's start any conversation about tape with a groan and eye rolling, which tends to be an attitude that keeps finite resources from being heavily invested in them. So, people will buy 10+Gbit links between their data centers (which sometimes tend to be shockingly inexpensive) rather than spend $10k on a tape library and a fedex account.