4 ms·

Buy servers if you have stable workloads, otherwide rent virtual machines in the cloud.

wenc 8y ago
The article also assumes 100% utilization on the cloud. I wonder if continual GPU-based training of DNN models is perhaps a fairly circumscribed use case? Most DNN model training workloads are lumpy and transient.
- ramraj07 8y ago
  Practically though the moment you use an instance more than 50% of the time AWS incentivizes buying the annual plan.
NotAnEconomist 8y ago
If I'm reading the chart right, GPU servers on AWS are cheaper if you utilize them less than 65-75% of the time, and turn them off when not in use.
- bithavoc 8y ago
  I think so, if you can afford turning off a server during the night then it makes sense to take advantage of some sort of hourly billing which is what most clouds offer by default.
  - deleted 8y ago
    [deleted]
  - stephenbez 8y ago
    EC2 has per-second billing: https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/ https://aws.amazon.com/blogs/aws/new-per-second-billing-for-...
rocky1138 8y ago
Using spot instances on AWS is a good way to save money on infrequent tasks, as well.
vidarh 8y ago
I'd add to that: Most people have far more stable workloads than they like to think. "Everyone" seems to think they have massively spiky loads, and almost nobody does. There may often be demands that could spike (e.g. because "someone" has not randomized the start times for batch maintenance processes through the night but start them all at the same time), but more often than not the spikiest demand comes from batch workloads where nobody cares if it takes a bit longer. In practice when constrained that often simply smooths out the overall capacity use. (of course that does not mean that there aren't people with spiky enough workloads for it to make sense to put at least some of their workloads on VMs for cost reasons; but measure first)
- Dunedan 8y ago
  Is that really the case? So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. It doesn't even matter if it's a globally used product, as some markets are usually stronger than others (e.g. US vs. Asia). So you have some base load, which is relatively stable, but on top of that daily or even more frequent "spikes". For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. That's something you probably wouldn't do with dedicated servers, simply because it'd be too expensive. Instead with dedicated servers you'd optimize for utilization instead of speed. When you only pay for what you use, you don't have that limitation and can fire up as many instances as it makes sense. That's actually a pretty nice benefit. And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code).
  - vidarh 8y ago
    > So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. Yes, but rarely big enough to allow for long enough periods of spinning down instances to be cheaper than dedicated hardware. Consider that (while not really a suitable example, since it's a GPU instance) the costs in the article are only that low for AWS because it's a reserved instance, where the savings of not using it are much lower than for regular instances. I've seen setups where we were close to considering using AWS for peak load, which would have been far cheaper than going all AWS, but when we did the calculations we needed peaks lasting shorter than 6-8 hours for it to be worthwhile vs. new hardware, and the cheaper alternative was simply leaving older machines in our racks past the 3 year depreciation schedule - it's not "free", they took up rack space and cost in terms of power, but it was cheaper to just let them stay in rotation until they failed or we needed the space for a faster machine, than it was to rent extra capacity. Scaling up with cloud services cost-effectively for daily peaks is tricky, because you're still going to be scaling ahead of use and most of your "peaker instances" are not going to have average utilization anywhere near 100% when factoring in ramping up/down for setups that aren't huge, because you still have lead times before an instance is maxed out, or as load is dropping, and you need to factor in that lower utilization in your cost estimates. That's not to say it's not worth considering, not least because having the ability to do so lets you load your dedicated equipment to much higher utilization rates because you're able to respond to spikes faster than you can add servers. > For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. You might. But my experience is that over a couple of decades of working with various types of online services, it's very rare for those types of workloads to move the needle much in terms of overall compute needs. There certainly are exceptions, and machine learning workloads might well make them more common, but they're comparatively rare in most companies. Keep in mind that while you may want capacity to handle spikes, in most such setups you also want capacity to handle loads that can run off peak. But by all means: If you do have needs that require those kind of big spikes that you can't accommodate within you base capacity, then use cloud instances for that. They do have their place. > And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code). In practice I've never seen dramatic enough changes anywhere for this to be a real concern for most people, as massive drop-offs tends to be rare, and tends to be eaten up by subsequent growth in a matter of months unless you're in big enough problems for it not to matter. If it's a potentially real concern to you, then rent dedicated servers by the month for some of the workload - it is typically still far cheaper than AWS, though not as much as leasing or buying - or do use a cloud provider for some of your peak load as an "insurance". The big takeaway is not that you shouldn't use cloud services, but that you should know your workload and do the cost calculations.
sabalaba 8y ago
I should probably change the first sentence in the article to this.

wenc 8y ago

The article also assumes 100% utilization on the cloud. I wonder if continual GPU-based training of DNN models is perhaps a fairly circumscribed use case? Most DNN model training workloads are lumpy and transient.

ramraj07 8y ago
Practically though the moment you use an instance more than 50% of the time AWS incentivizes buying the annual plan.

NotAnEconomist 8y ago

If I'm reading the chart right, GPU servers on AWS are cheaper if you utilize them less than 65-75% of the time, and turn them off when not in use.

bithavoc 8y ago
I think so, if you can afford turning off a server during the night then it makes sense to take advantage of some sort of hourly billing which is what most clouds offer by default.
- deleted 8y ago
  [deleted]
- stephenbez 8y ago
  EC2 has per-second billing: https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/ https://aws.amazon.com/blogs/aws/new-per-second-billing-for-...

rocky1138 8y ago

Using spot instances on AWS is a good way to save money on infrequent tasks, as well.

vidarh 8y ago

I'd add to that: Most people have far more stable workloads than they like to think. "Everyone" seems to think they have massively spiky loads, and almost nobody does. There may often be demands that could spike (e.g. because "someone" has not randomized the start times for batch maintenance processes through the night but start them all at the same time), but more often than not the spikiest demand comes from batch workloads where nobody cares if it takes a bit longer. In practice when constrained that often simply smooths out the overall capacity use. (of course that does not mean that there aren't people with spiky enough workloads for it to make sense to put at least some of their workloads on VMs for cost reasons; but measure first)

Dunedan 8y ago
Is that really the case? So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. It doesn't even matter if it's a globally used product, as some markets are usually stronger than others (e.g. US vs. Asia). So you have some base load, which is relatively stable, but on top of that daily or even more frequent "spikes". For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. That's something you probably wouldn't do with dedicated servers, simply because it'd be too expensive. Instead with dedicated servers you'd optimize for utilization instead of speed. When you only pay for what you use, you don't have that limitation and can fire up as many instances as it makes sense. That's actually a pretty nice benefit. And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code).
- vidarh 8y ago
  > So far all companies I worked for had a wave-like pattern for customer generated workloads, simply because of the locality of the customers. Yes, but rarely big enough to allow for long enough periods of spinning down instances to be cheaper than dedicated hardware. Consider that (while not really a suitable example, since it's a GPU instance) the costs in the article are only that low for AWS because it's a reserved instance, where the savings of not using it are much lower than for regular instances. I've seen setups where we were close to considering using AWS for peak load, which would have been far cheaper than going all AWS, but when we did the calculations we needed peaks lasting shorter than 6-8 hours for it to be worthwhile vs. new hardware, and the cheaper alternative was simply leaving older machines in our racks past the 3 year depreciation schedule - it's not "free", they took up rack space and cost in terms of power, but it was cheaper to just let them stay in rotation until they failed or we needed the space for a faster machine, than it was to rent extra capacity. Scaling up with cloud services cost-effectively for daily peaks is tricky, because you're still going to be scaling ahead of use and most of your "peaker instances" are not going to have average utilization anywhere near 100% when factoring in ramping up/down for setups that aren't huge, because you still have lead times before an instance is maxed out, or as load is dropping, and you need to factor in that lower utilization in your cost estimates. That's not to say it's not worth considering, not least because having the ability to do so lets you load your dedicated equipment to much higher utilization rates because you're able to respond to spikes faster than you can add servers. > For non-customer-driven workloads you even might want to have spikes. Take big data pipelines for example. While you could schedule them so that they're not spiky, you might benefit from running them as fast a possible to get your results earlier. You might. But my experience is that over a couple of decades of working with various types of online services, it's very rare for those types of workloads to move the needle much in terms of overall compute needs. There certainly are exceptions, and machine learning workloads might well make them more common, but they're comparatively rare in most companies. Keep in mind that while you may want capacity to handle spikes, in most such setups you also want capacity to handle loads that can run off peak. But by all means: If you do have needs that require those kind of big spikes that you can't accommodate within you base capacity, then use cloud instances for that. They do have their place. > And don't forget changing workloads over the course of multiple months. The calculation looks completely different if you buy a dozen of servers now, but have your computing requirements change in a few months, so that you don't need them all anymore (reasons for such are manifold, but could be as simple as optimizations to your code). In practice I've never seen dramatic enough changes anywhere for this to be a real concern for most people, as massive drop-offs tends to be rare, and tends to be eaten up by subsequent growth in a matter of months unless you're in big enough problems for it not to matter. If it's a potentially real concern to you, then rent dedicated servers by the month for some of the workload - it is typically still far cheaper than AWS, though not as much as leasing or buying - or do use a cloud provider for some of your peak load as an "insurance". The big takeaway is not that you shouldn't use cloud services, but that you should know your workload and do the cost calculations.

sabalaba 8y ago

I should probably change the first sentence in the article to this.