6 ms·

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical q

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amount of money reducing latency :) We're not a high frequency trading firm or anything, but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems. > Reliability is "solved" at low levels To whatever extent this may be true, it's certainly not true for cloud providers. One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems.

HatchedLake721 6y ago
Seems you’re trying to bring your on-premise structure and concepts to the cloud, that won’t work. EC2 instances are cattle, not pets. I believe I saw a slide that average lifespan of an EC2 instance at Netflix is 14 minutes. I’m not necessarily saying cloud will work for your setup, but you can’t compare like that.
- pgwhalen 6y ago
  It seems to me like they're not trying to bring on-prem to the cloud, and that's very much the point.
- toomuchtodo 6y ago
  Most businesses are not Netflix, where most of the work is done by their OpenConnect CDN appliances serving video content outside of cloud providers.
tstrimple 6y ago
> How much would it cost to move this across boxes in EC2? Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). You're not going to have a successful cloud experience unless you build your applications in a cloud suitable way. This means not all legacy applications are a good fit for the public cloud. Most companies really embracing the cloud are mitigating those risks by distributing workloads across multiple instances so you don't care if any one needs to be restarted, especially within a planned window. > Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems. Are these inter-region network dropouts or between the internet and the cloud data center? You're not going to be relying on a public internet connection to the cloud for critical workloads. All that being said, there are plenty of workloads which I don't think fit well in the cloud operating model. You may very well have one of them.
- iampims 6y ago
  You pay for cross-AZ Traffic in AWS, and that adds up really fast.
  - tstrimple 6y ago
    Yes. You've got to be aware of where those boundaries are when adopting the cloud and the cost information around these cases are inadequate at best. Too many people get surprise bills.
    - hinkley 6y ago
      The first step is to stop blaming the victim. It's nobody's fault that the billing structure at Amazon is so complicated and confusing. Except Amazon's.
  - Wintereise 6y ago
resonator 6y ago
> How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. Data transfer between instances in the same AZ is free. If the data crosses AZs, you're changed $0.01 per GB in both directions. This is for instances on a VPC. I think the pricing model is different for classic EC2. There are some exceptions like all traffic between EC2 and ALBs being free. Edit: Pricing is described at https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
- takeda 6y ago
  and if you go for resilience you will do a lot of that cross AZ talking, unless you also design your apps to know the infrastructure and consider that when communicating (most companies don't do it)
H8crilA 6y ago
> A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. I think everyone does something of this sort nowadays, that's why networking is ~free within data centers :) > but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems That speaks to me. You will always be just the n-th client unless you own the cross-datacenter data links (i.e. have full autonomy on deciding the priority of the traffic). It's similar to the covid provisioning problems you had mentioned. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. Yeah like others pointed out - that's just what "cloud" is, and is generally a good idea. You're supposed to handle a certain % of your machines going dark without a warning without violating any SLO (or even worse, certain % of your machines "pretending" they're up but actually being ridiculously slow for this or that reason; and don't even get me started on CPU/RAM bitflips). It sounds to me that you run an extremely highly sensitive service, something for which paying for true ownership of the hardware just makes sense to remove those kinds of risks that most services don't care about. At the end of the day "cloud" is a shared resource, and no resource separation efforts will be 100% effective.
kelnos 6y ago
> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). It's interesting that you say this, because I think this just boils down to how you treat failures. Boxes will fail. It's inevitable, even if you run your own hardware. We treat failure as an every-day event and have set up our systems so it doesn't matter. One box fails, an automated system notices and brings up another to replace it, while the remaining boxes take slightly more load for a few minutes. Sure, that kind of failure tolerance doesn't come for free. But, again, you're going to have failures, no matter how much work you put into reliability (which also doesn't come for free!).

HatchedLake721 6y ago

Seems you’re trying to bring your on-premise structure and concepts to the cloud, that won’t work. EC2 instances are cattle, not pets. I believe I saw a slide that average lifespan of an EC2 instance at Netflix is 14 minutes. I’m not necessarily saying cloud will work for your setup, but you can’t compare like that.

pgwhalen 6y ago
It seems to me like they're not trying to bring on-prem to the cloud, and that's very much the point.
toomuchtodo 6y ago
Most businesses are not Netflix, where most of the work is done by their OpenConnect CDN appliances serving video content outside of cloud providers.

tstrimple 6y ago

> How much would it cost to move this across boxes in EC2? Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). You're not going to have a successful cloud experience unless you build your applications in a cloud suitable way. This means not all legacy applications are a good fit for the public cloud. Most companies really embracing the cloud are mitigating those risks by distributing workloads across multiple instances so you don't care if any one needs to be restarted, especially within a planned window. > Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems. Are these inter-region network dropouts or between the internet and the cloud data center? You're not going to be relying on a public internet connection to the cloud for critical workloads. All that being said, there are plenty of workloads which I don't think fit well in the cloud operating model. You may very well have one of them.

iampims 6y ago
You pay for cross-AZ Traffic in AWS, and that adds up really fast.
- tstrimple 6y ago
  Yes. You've got to be aware of where those boundaries are when adopting the cloud and the cost information around these cases are inadequate at best. Too many people get surprise bills.
  - hinkley 6y ago
    The first step is to stop blaming the victim. It's nobody's fault that the billing structure at Amazon is so complicated and confusing. Except Amazon's.
- Wintereise 6y ago

resonator 6y ago

> How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. Data transfer between instances in the same AZ is free. If the data crosses AZs, you're changed $0.01 per GB in both directions. This is for instances on a VPC. I think the pricing model is different for classic EC2. There are some exceptions like all traffic between EC2 and ALBs being free. Edit: Pricing is described at https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

takeda 6y ago
and if you go for resilience you will do a lot of that cross AZ talking, unless you also design your apps to know the infrastructure and consider that when communicating (most companies don't do it)

H8crilA 6y ago

> A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. I think everyone does something of this sort nowadays, that's why networking is ~free within data centers :) > but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems That speaks to me. You will always be just the n-th client unless you own the cross-datacenter data links (i.e. have full autonomy on deciding the priority of the traffic). It's similar to the covid provisioning problems you had mentioned. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. Yeah like others pointed out - that's just what "cloud" is, and is generally a good idea. You're supposed to handle a certain % of your machines going dark without a warning without violating any SLO (or even worse, certain % of your machines "pretending" they're up but actually being ridiculously slow for this or that reason; and don't even get me started on CPU/RAM bitflips). It sounds to me that you run an extremely highly sensitive service, something for which paying for true ownership of the hardware just makes sense to remove those kinds of risks that most services don't care about. At the end of the day "cloud" is a shared resource, and no resource separation efforts will be 100% effective.

kelnos 6y ago

> One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). It's interesting that you say this, because I think this just boils down to how you treat failures. Boxes will fail. It's inevitable, even if you run your own hardware. We treat failure as an every-day event and have set up our systems so it doesn't matter. One box fails, an automated system notices and brings up another to replace it, while the remaining boxes take slightly more load for a few minutes. Sure, that kind of failure tolerance doesn't come for free. But, again, you're going to have failures, no matter how much work you put into reliability (which also doesn't come for free!).