4 ms·

But it was always like that for HDDs. You always had to assume no change in seek time with more capacity. And it was always wrong to treat HDDs as random IO dev

But it was always like that for HDDs. You always had to assume no change in seek time with more capacity. And it was always wrong to treat HDDs as random IO devices with all the random writes and b-trees. At least now there is a pressure on everyone involved to do it right.

notacoward 8y ago
That's a completely different issue. Yes, it's important to do disk accesses right. That was even more true when I started doing this stuff in 1990 than it is now, because disks back then didn't have big caches or fancy controllers, and RAID wasn't a thing. My point is that even if you do everything right it's getting harder and harder for a disk-based system to keep up with I/O demand. That means more and more gyrations to keep the demand away from those disk-based systems, which means more overall system complexity. In particular, more places for inconsistency to creep in. As though we didn't have enough to deal with making these things distributed.
- zzzcpan 8y ago
  I get it, but take RAID for example. It always presumed weird things. That you either have enough performance and not enough load to be able to check disks or rebuild it without much degradation or that you are able to copy an entire disk in like a few hours or that an error means you need to replace the disk. These assumptions break if you don't forget that capacity increases, but seek time stays the same. Obviously there will be some inherent complexity dealing with that, but not really more complexity, it's just different, maybe proper for once.
  - notacoward 8y ago
    I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next one was scheduled. Ah, the good old days. That was the easy case. The system I work on today has a dozen such maintenance activities going on all the time. Making sure they all complete in reasonable time, without affecting the endless stream of new user I/O too much, requires increasingly sophisticated scheduling as the gap gets wider. That's a lot of complexity right there. On top of that, there's this little formula: load = warmth * capacity Warmth is effectively constant for relevant time scales, so increasing capacity means increasing load. Scaling out doesn't help in this case, not even with perfect linear scaling, because every bit of added capability brings added capacity and load with it. Until you run out of users, I guess, but those seem to be infinite. ;) Thus, systems with the same basic architecture get more and more overloaded as per-disk capacity increases, until they become useless. The only escape is to change the architecture, inevitably toward greater complexity (e.g. cache layers and burst buffers). None of this would be necessary with a better capacity/performance ratio. Believe me, nobody wants to make these systems more complicated. But every time that gap gets bigger, there will likely be a new increment of complexity to deal with it I don't see how that's "proper" or "where it belongs" or any such. It's not that all such systems were poorly designed for the hardware as it was when they were developed. It's that even the best designs have to keep adapting. The demise of Moore's law and the ever increasing number of cores per die or per system have increased complexity in the compute domain. The capacity/performance gap is the storage equivalent.
    - zzzcpan 8y ago
      Cheap cold storage will find its use. So I think it's reasonable to expect the gap to increase for like 10x more in the future, before SSDs overtake storage completely. Have to plan at least for that.

notacoward 8y ago

That's a completely different issue. Yes, it's important to do disk accesses right. That was even more true when I started doing this stuff in 1990 than it is now, because disks back then didn't have big caches or fancy controllers, and RAID wasn't a thing. My point is that even if you do everything right it's getting harder and harder for a disk-based system to keep up with I/O demand. That means more and more gyrations to keep the demand away from those disk-based systems, which means more overall system complexity. In particular, more places for inconsistency to creep in. As though we didn't have enough to deal with making these things distributed.

zzzcpan 8y ago
I get it, but take RAID for example. It always presumed weird things. That you either have enough performance and not enough load to be able to check disks or rebuild it without much degradation or that you are able to copy an entire disk in like a few hours or that an error means you need to replace the disk. These assumptions break if you don't forget that capacity increases, but seek time stays the same. Obviously there will be some inherent complexity dealing with that, but not really more complexity, it's just different, maybe proper for once.
- notacoward 8y ago
  I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next one was scheduled. Ah, the good old days. That was the easy case. The system I work on today has a dozen such maintenance activities going on all the time. Making sure they all complete in reasonable time, without affecting the endless stream of new user I/O too much, requires increasingly sophisticated scheduling as the gap gets wider. That's a lot of complexity right there. On top of that, there's this little formula: load = warmth * capacity Warmth is effectively constant for relevant time scales, so increasing capacity means increasing load. Scaling out doesn't help in this case, not even with perfect linear scaling, because every bit of added capability brings added capacity and load with it. Until you run out of users, I guess, but those seem to be infinite. ;) Thus, systems with the same basic architecture get more and more overloaded as per-disk capacity increases, until they become useless. The only escape is to change the architecture, inevitably toward greater complexity (e.g. cache layers and burst buffers). None of this would be necessary with a better capacity/performance ratio. Believe me, nobody wants to make these systems more complicated. But every time that gap gets bigger, there will likely be a new increment of complexity to deal with it I don't see how that's "proper" or "where it belongs" or any such. It's not that all such systems were poorly designed for the hardware as it was when they were developed. It's that even the best designs have to keep adapting. The demise of Moore's law and the ever increasing number of cores per die or per system have increased complexity in the compute domain. The capacity/performance gap is the storage equivalent.
  - zzzcpan 8y ago
    Cheap cold storage will find its use. So I think it's reasonable to expect the gap to increase for like 10x more in the future, before SSDs overtake storage completely. Have to plan at least for that.