4 ms·

Back when I used AWS, I went the reserved instances route. The pricing is pretty okay, and you are guaranteed to have machines at busy periods of the day. The

Back when I used AWS, I went the reserved instances route. The pricing is pretty okay, and you are guaranteed to have machines at busy periods of the day. The problem with spot autoscaling is that if everyone does it, it stops working. Everyone in us-east-1 gets most of their traffic from 9am EST to 5pm EST, because everyone hosts close to their users, their users are human, and humans are diurnal. Most "batch" workloads that people have also follow the same cycle; they're working on something in the office and want results now, not tomorrow morning. So they run their batch jobs during the day. If you can figure out how to get your traffic spikes from 9pm until 5am, spot instances are going to be great -- millions of CPUs are sitting idle waiting for your novel workload. But if your customers are working 9-5 jobs like you, and you care about latency enough to host close to them... you're competing for instances with every other computer user in the region.

lukasLansky 7y ago
Most of batch workloads are not that latency sensitive. You can send them to the other side of the world as everyone is sleeping there right now.
- satanspastaroll 7y ago
  A thing to keep in mind is AWS's major outobound traffic costs, at $90/TB. So taxiing the data oversea might not be affordable either
  - oblio 7y ago
    Even for inter-AWS region transfers?
    - mi100hael 7y ago
      It depends on how the data is being transferred. In general, the answer is yes because you'd be using public IPs. If you're transferring between EC2 instances in peered VPCs meaning you can still use private IPs, it's more like $20/TB. https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ https://datapath.io/resources/blog/what-are-aws-data-transfe...
- jrockway 7y ago
  That's not been my experience, actually. The first CI service I used was hosted entirely in Europe, and we needed to ssh in to debug something, the keystroke latency was maddening. We eventually unsubscribed and just bought a big EC2 instance in us-east with Jenkins running on it. It cost approximately 100x more, but our productivity was high and frustration low. Well worth. I personally think it will breed huge organizational problems if things like CI are slow. "I'll get a cup of coffee while this runs" and then you come back and forget what you were going to release. Soon it becomes "let's get another change into this build before we release" and then it's "well, it's been six months since we've released anything, what do we do." You have to start fast and stay fast if you want to keep developers productive. So saving a couple bucks on computers that are half a world away can end up being a huge expense if you're not careful. As other comments mention, you also have to be careful about transfer costs. In the CI case, getting your source code into the CI server is cheap, but getting the containers out is going to cost you, especially if you don't make an effort to optimize them. For batch data processing jobs, the same applies; getting the result out is cheap, but getting the data in is going to be a lot of transfer. (If you were using Small Data, you could just run the job on your laptop, after all.) The speed of computers half a world away is not great either. I remember updating some Samsung drivers once, which were served out of a Korean AWS region instead of CloudFront... and the downloads were glacially slow. Their website is the same way. I couldn't believe how a multinational corporation could push bits at me so slowly. When you're reading their documentation all day, or tweaking drivers, you notice it, and you start to think "next time I'm going to buy Intel". (Compare Samsung's SSD website with McMaster-Carr's website. What site do you hope to interact with again in the future?) Anyway, you get a bill for compute resources, and you don't get a bill for unhappy employees context-switching all day, so I see why people want to craft clever schemes to save pennies on their compute costs. But be careful. Not every cost is charged directly to your credit card.
  - heartbeats 7y ago
    What about using something like mosh for latency?

lukasLansky 7y ago

Most of batch workloads are not that latency sensitive. You can send them to the other side of the world as everyone is sleeping there right now.

satanspastaroll 7y ago
A thing to keep in mind is AWS's major outobound traffic costs, at $90/TB. So taxiing the data oversea might not be affordable either
- oblio 7y ago
  Even for inter-AWS region transfers?
  - mi100hael 7y ago
    It depends on how the data is being transferred. In general, the answer is yes because you'd be using public IPs. If you're transferring between EC2 instances in peered VPCs meaning you can still use private IPs, it's more like $20/TB. https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ https://datapath.io/resources/blog/what-are-aws-data-transfe...
jrockway 7y ago
That's not been my experience, actually. The first CI service I used was hosted entirely in Europe, and we needed to ssh in to debug something, the keystroke latency was maddening. We eventually unsubscribed and just bought a big EC2 instance in us-east with Jenkins running on it. It cost approximately 100x more, but our productivity was high and frustration low. Well worth. I personally think it will breed huge organizational problems if things like CI are slow. "I'll get a cup of coffee while this runs" and then you come back and forget what you were going to release. Soon it becomes "let's get another change into this build before we release" and then it's "well, it's been six months since we've released anything, what do we do." You have to start fast and stay fast if you want to keep developers productive. So saving a couple bucks on computers that are half a world away can end up being a huge expense if you're not careful. As other comments mention, you also have to be careful about transfer costs. In the CI case, getting your source code into the CI server is cheap, but getting the containers out is going to cost you, especially if you don't make an effort to optimize them. For batch data processing jobs, the same applies; getting the result out is cheap, but getting the data in is going to be a lot of transfer. (If you were using Small Data, you could just run the job on your laptop, after all.) The speed of computers half a world away is not great either. I remember updating some Samsung drivers once, which were served out of a Korean AWS region instead of CloudFront... and the downloads were glacially slow. Their website is the same way. I couldn't believe how a multinational corporation could push bits at me so slowly. When you're reading their documentation all day, or tweaking drivers, you notice it, and you start to think "next time I'm going to buy Intel". (Compare Samsung's SSD website with McMaster-Carr's website. What site do you hope to interact with again in the future?) Anyway, you get a bill for compute resources, and you don't get a bill for unhappy employees context-switching all day, so I see why people want to craft clever schemes to save pennies on their compute costs. But be careful. Not every cost is charged directly to your credit card.
- heartbeats 7y ago
  What about using something like mosh for latency?