3 ms·
I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next o
by notacoward 8y ago
I'd say it's a lot more complexity. The first way most people became aware of the capacity/performance gaps is that a backup couldn't complete before the next one was scheduled. Ah, the good old days. That was the easy case. The system I work on today has a dozen such maintenance activities going on all the time. Making sure they all complete in reasonable time, without affecting the endless stream of new user I/O too much, requires increasingly sophisticated scheduling as the gap gets wider. That's a lot of complexity right there. On top of that, there's this little formula:
load = warmth * capacity
Warmth is effectively constant for relevant time scales, so increasing capacity means increasing load. Scaling out doesn't help in this case, not even with perfect linear scaling, because every bit of added capability brings added capacity and load with it. Until you run out of users, I guess, but those seem to be infinite. ;)
Thus, systems with the same basic architecture get more and more overloaded as per-disk capacity increases, until they become useless. The only escape is to change the architecture, inevitably toward greater complexity (e.g. cache layers and burst buffers). None of this would be necessary with a better capacity/performance ratio. Believe me, nobody wants to make these systems more complicated. But every time that gap gets bigger, there will likely be a new increment of complexity to deal with it
I don't see how that's "proper" or "where it belongs" or any such. It's not that all such systems were poorly designed for the hardware as it was when they were developed. It's that even the best designs have to keep adapting. The demise of Moore's law and the ever increasing number of cores per die or per system have increased complexity in the compute domain. The capacity/performance gap is the storage equivalent.
- zzzcpan 8y agoCheap cold storage will find its use. So I think it's reasonable to expect the gap to increase for like 10x more in the future, before SSDs overtake storage completely. Have to plan at least for that.