3 ms·

Is that really the case? So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the cust

Is that really the case? So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. It doesn't even matter if it's a globally used product, as some markets are usually stronger than others (e.g. US vs. Asia). So you have some base load, which is relatively stable, but on top of that daily or even more frequent "spikes". For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. That's something you probably wouldn't do with dedicated servers, simply because it'd be too expensive. Instead with dedicated servers you'd optimize for utilization instead of speed. When you only pay for what you use, you don't have that limitation and can fire up as many instances as it makes sense. That's actually a pretty nice benefit. And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code).

vidarh 8y ago
> So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. Yes, but rarely big enough to allow for long enough periods of spinning down instances to be cheaper than dedicated hardware. Consider that (while not really a suitable example, since it's a GPU instance) the costs in the article are only that low for AWS because it's a reserved instance, where the savings of not using it are much lower than for regular instances. I've seen setups where we were close to considering using AWS for peak load, which would have been far cheaper than going all AWS, but when we did the calculations we needed peaks lasting shorter than 6-8 hours for it to be worthwhile vs. new hardware, and the cheaper alternative was simply leaving older machines in our racks past the 3 year depreciation schedule - it's not "free", they took up rack space and cost in terms of power, but it was cheaper to just let them stay in rotation until they failed or we needed the space for a faster machine, than it was to rent extra capacity. Scaling up with cloud services cost-effectively for daily peaks is tricky, because you're still going to be scaling ahead of use and most of your "peaker instances" are not going to have average utilization anywhere near 100% when factoring in ramping up/down for setups that aren't huge, because you still have lead times before an instance is maxed out, or as load is dropping, and you need to factor in that lower utilization in your cost estimates. That's not to say it's not worth considering, not least because having the ability to do so lets you load your dedicated equipment to much higher utilization rates because you're able to respond to spikes faster than you can add servers. > For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. You might. But my experience is that over a couple of decades of working with various types of online services, it's very rare for those types of workloads to move the needle much in terms of overall compute needs. There certainly are exceptions, and machine learning workloads might well make them more common, but they're comparatively rare in most companies. Keep in mind that while you may want capacity to handle spikes, in most such setups you also want capacity to handle loads that can run off peak. But by all means: If you do have needs that require those kind of big spikes that you can't accommodate within you base capacity, then use cloud instances for that. They do have their place. > And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code). In practice I've never seen dramatic enough changes anywhere for this to be a real concern for most people, as massive drop-offs tends to be rare, and tends to be eaten up by subsequent growth in a matter of months unless you're in big enough problems for it not to matter. If it's a potentially real concern to you, then rent dedicated servers by the month for some of the workload - it is typically still far cheaper than AWS, though not as much as leasing or buying - or do use a cloud provider for some of your peak load as an "insurance". The big takeaway is not that you shouldn't use cloud services, but that you should know your workload and do the cost calculations.

vidarh 8y ago

> So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. Yes, but rarely big enough to allow for long enough periods of spinning down instances to be cheaper than dedicated hardware. Consider that (while not really a suitable example, since it's a GPU instance) the costs in the article are only that low for AWS because it's a reserved instance, where the savings of not using it are much lower than for regular instances. I've seen setups where we were close to considering using AWS for peak load, which would have been far cheaper than going all AWS, but when we did the calculations we needed peaks lasting shorter than 6-8 hours for it to be worthwhile vs. new hardware, and the cheaper alternative was simply leaving older machines in our racks past the 3 year depreciation schedule - it's not "free", they took up rack space and cost in terms of power, but it was cheaper to just let them stay in rotation until they failed or we needed the space for a faster machine, than it was to rent extra capacity. Scaling up with cloud services cost-effectively for daily peaks is tricky, because you're still going to be scaling ahead of use and most of your "peaker instances" are not going to have average utilization anywhere near 100% when factoring in ramping up/down for setups that aren't huge, because you still have lead times before an instance is maxed out, or as load is dropping, and you need to factor in that lower utilization in your cost estimates. That's not to say it's not worth considering, not least because having the ability to do so lets you load your dedicated equipment to much higher utilization rates because you're able to respond to spikes faster than you can add servers. > For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. You might. But my experience is that over a couple of decades of working with various types of online services, it's very rare for those types of workloads to move the needle much in terms of overall compute needs. There certainly are exceptions, and machine learning workloads might well make them more common, but they're comparatively rare in most companies. Keep in mind that while you may want capacity to handle spikes, in most such setups you also want capacity to handle loads that can run off peak. But by all means: If you do have needs that require those kind of big spikes that you can't accommodate within you base capacity, then use cloud instances for that. They do have their place. > And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code). In practice I've never seen dramatic enough changes anywhere for this to be a real concern for most people, as massive drop-offs tends to be rare, and tends to be eaten up by subsequent growth in a matter of months unless you're in big enough problems for it not to matter. If it's a potentially real concern to you, then rent dedicated servers by the month for some of the workload - it is typically still far cheaper than AWS, though not as much as leasing or buying - or do use a cloud provider for some of your peak load as an "insurance". The big takeaway is not that you shouldn't use cloud services, but that you should know your workload and do the cost calculations.