3 ms·

This is not at all a good way to compare. Engineering hours: there's a good chance you pay for that once, and that solves the problem. 10 SSDs: will require r

This is not at all a good way to compare. Engineering hours: there's a good chance you pay for that once, and that solves the problem. 10 SSDs: will require rack space, electricity, PCIe slots, timely replacement, management software... most of these expenses will be recurring expenses. If done once, sometimes the existing infrastructure can amortize these expenses (i.e. you might have already had empty space in a rack, you might have already had spare PCIe slots, etc.), but amortization will only work in small numbers. Another aspect of this trade-off: systems inevitably lose performance per unit of equipment as they grow due to management expenses and increased latency. So, if you keep solving problems by growing the system, overall, the system will become more and more "sluggish" until it becomes unserviceable. On the other hand, solutions which minimize the number of system resources necessary to accomplish a task increase overall performance per unit. In other words, create a higher-quality system, which is an asset in its own right.

ilyt 3y ago
I mean I'm also proponent of "doing it right" most of the time, but throwing hardware at the problem is mighty fine solution in sub-TB areas > 10 SSDs: will require rack space, electricity, PCIe slots, timely replacement, management software... most of these expenses will be recurring expenses. If done once, sometimes the existing infrastructure can amortize these expenses (i.e. you might have already had empty space in a rack, you might have already had spare PCIe slots, etc.), but amortization will only work in small numbers. 10NVMe SSDs fit into 1U. The cloud always comes out terrible on hardware. The cloud is not selling hardware, it's terrible to buy any hardware in cloud, it sells spike capacity, and it sells software running on it. There are many services in cloud that would be a lot of work to create (even if you need just small part of functionality) on-site but if you don't use it and just need to throw raw hardware at the problem consistently, a rack with hardware will be far cheaper solution even after incurring the engineering cost to deploy and automate it. We have 7 racks and time to manage hardware is some minuscule fraction, and time to manage automation on it is still small part of our time, and we're just ops team of 3.

ilyt 3y ago

I mean I'm also proponent of "doing it right" most of the time, but throwing hardware at the problem is mighty fine solution in sub-TB areas > 10 SSDs: will require rack space, electricity, PCIe slots, timely replacement, management software... most of these expenses will be recurring expenses. If done once, sometimes the existing infrastructure can amortize these expenses (i.e. you might have already had empty space in a rack, you might have already had spare PCIe slots, etc.), but amortization will only work in small numbers. 10NVMe SSDs fit into 1U. The cloud always comes out terrible on hardware. The cloud is not selling hardware, it's terrible to buy any hardware in cloud, it sells spike capacity, and it sells software running on it. There are many services in cloud that would be a lot of work to create (even if you need just small part of functionality) on-site but if you don't use it and just need to throw raw hardware at the problem consistently, a rack with hardware will be far cheaper solution even after incurring the engineering cost to deploy and automate it. We have 7 racks and time to manage hardware is some minuscule fraction, and time to manage automation on it is still small part of our time, and we're just ops team of 3.