6 ms·

This excessive inter-AZ data transfer pricing is distorting engineering best practices. It _should_ be cheap to operate HA systems across 2-3 AZs, but because o

This excessive inter-AZ data transfer pricing is distorting engineering best practices. It _should_ be cheap to operate HA systems across 2-3 AZs, but because of this price distortion on inter-AZ traffic charges, we lean towards designs that either silo data within an AZ, or that leverage S3 or other hosted solutions as a sort of accounting workaround (i.e. there are no data transfer charges to read/write an S3 bucket from any AZ in the same region). While AWS egress pricing gets a lot of attention, I think that the high cost of inter-AZ traffic is much less defensible. This is transfer on short fat pipes completely owned by Amazon. And at $0.01/GB, that's 2~10X what smaller providers charge for _internet_ egress.

hipadev23 2y ago
I assume it’s to discourage people from architecting designs that abuse the network. A good example would be collecting every single metric, every second, across every instance for no real business reason.
- thayne 2y ago
  Or maybe it is price discrimination. A way to extract more money from customers that need higher availability and probably have higher budgets.
  - hipadev23 2y ago
    Price discrimination is when you charge different amounts for the same thing to different customers. And usually the difference in those prices are not made apparent. Like when travel websites quote iOS users more than Android because they generally can afford to pay more. This is just regular ole pricing.
    - thayne 2y ago
      So what is the correct term for "charge an extremely high markup for a feature that some, but not all, of your customers need"?
      - spondylosaurus 2y ago
        Price gouging?
      - jchanimal 2y ago
        I came here to say the same thing. When you’re selling cloud services, the hardest thing to do is segment your customers by willingness to pay. Cross AZ traffic is exactly the sort of thing companies with budgets need, that small projects don’t.
      - mcmcmc 2y ago
        Supply and demand
    - hansvm 2y ago
      It's a bit of a mix, but price discrimination isn't far off. It's like the SSO tax; all organizations are paying for effectively the same service, but the provider has found a minor way to cripple the service that selectively targets people who can afford to pay more. If we want to call this just regular ole pricing, it's not a leap to call most textbook cases of price discrimination "regular ole pricing" as well. An online game charges more if your IP is from a certain geography? That's not discrimination; we've simply priced the product differently if you live in Silicon Valley; don't buy it it you don't want it.
      - hipadev23 2y ago
        https://en.wikipedia.org/wiki/Price_discrimination https://en.wikipedia.org/wiki/Price_discrimination Price discrimination has a clear definition. It’s not illegal in the US (when consumers are the victims anyway) but it has a clear meaning and you’re blurring the lines for I’m not sure what reason. Your example of a video game doing regional pricing is a perfect example of textbook price discrimination. Pricing a good or service at a level that inherently excludes those unwilling or unable to pay is: https://en.wikipedia.org/wiki/Excludability https://en.wikipedia.org/wiki/Excludability
        hansvm 2y ago
        Alright, let's take a look at that first link as if it's gospel. AWS charging excessively for inter-AZ networking is: 1. a microeconomic pricing strategy 2. where largely similar goods (AWS with or without substantial inter-AZ bandwidth) 3. are sold at different prices (excessive inter-AZ networking fees) to different buyers 4. based on perceived market segments (most customers don't need (or don't know they need till they're locked in) much inter-AZ bandwidth, but larger, richer corporations likely do) I'm not trying to blur the lines. On top of any juggling of our favorite sources of definitions, that particular pricing strategy has all the qualitative hallmarks of price discrimination. Everyone still buys AWS, most customers are unaffected by the lack of bulk inter-AZ bandwidth, and AWS can successfully charge much more to those who can afford to pay.
        thayne 2y ago
  - cowsandmilk 2y ago
    Sending traffic between AZs doesn’t necessarily improve availability and can decrease it. Each of your services can be multi-az, but have hosts that talk just to other endpoints in their AZ.
    - thayne 2y ago
      Unless your app is completely stateless, you will need some level of communication across AZs. And you often want cross-zone routing on your load balancers so that if you lose all the instances in one AZ traffic will still get routed to healthy instances.
  - KaiserPro 2y ago
    > Or maybe it is price discrimination. It very much is, because scaling bandwidth between phyical datacenters which are not located next to each other is very expensive. So pricing it means that people don't use it as much as if it was free.
    - mcmcmc 2y ago
      That’s not what price discrimination is
- koolba 2y ago
  You can still do that if you buffer it and push it to S3. Any AZ to S3 is free and the net result is the same.
KaiserPro 2y ago
I don't work for AWS However I do work for a company with >1 million servers. Scaling inter datacentre bandwidth is quite hard. Sure the datacentres might be geographically close, but laying network cables over distance is expensive. Moreover unless you spend uber millions, you're never going to get as much bandwidth as you have inside the datacentre. So you either apply hard limits per account, or price it so that people think twice about using it.
- themgt 2y ago
  OK but $10/TB has gotta be like >99% profit margin for AWS. After massively jacking up their prices, Hetzner internet egress is only €1/TB. Also AWS encourages / in some cases practically forces you to do multi-AZ. I remember switching to autoscaling spot instances to save a few bucks, then occasionally spot spinup would fail due to lack of availability within an AZ so I enabled multi-AZ spot. Then got hit with the inter-AZ bandwidth charges and wasn't actually saving any money vs single-AZ reserved. This was about the point I decided DIY Kubernetes was simpler to reason about.
  - everfrustrated 2y ago
    Apples and Oranges. Hetzner doesn't even have multiple AZ's by AWS's definition - all Hetzners DCs eg Falkenstein 1-14 would be the same AZ zone. AWS network is designed with a lot more internal capacity and reliability than Hetzner which costs a lot more - multiple uplinks to independent switches, etc. AWS is also buying current gen network gear which is much more pricey - Hetzner is mostly doing 1 Gig ports or 10 gig at a push which means they can get away with >10 year old switches (if you think they buy new switches I have a bridge you might be interested in buying). This costs at least an order of magnitude more.
    - iscoelho 2y ago
      I agree with this post that Hetzner is a bad example. They are focused on a budget deployment. I do not agree that a state-of-the-art high capacity deployment is as expensive as you think it is. If an organization pays MSRP on everything, has awful procurement with nonexistent negotiation, and multiple project failures, sure, maybe. In the real world though, we're not all working for the federal government (-:
timewizard 2y ago
> It _should_ be cheap to operate HA systems across 2-3 AZs, In the steady state. HA systems tend towards large data bursts when failures or upgrades occur. > And at $0.01/GB, that's 2~10X what smaller providers charge for _internet_ egress. It's a lower latency network with a high SLA and automatic credits if the SLA isn't maintained. I think the inter-AZ option provides a level of service that's much higher than what most people want or need. It might be nice if there was a "best effort" inter-AZ network. This would probably fit better with the synchronization methods built into most HA software anyways. So, to me, it's a good product, it's just designed for a very niche segment of the market and often mistaken for something more general than it actually is.

hipadev23 2y ago

I assume it’s to discourage people from architecting designs that abuse the network. A good example would be collecting every single metric, every second, across every instance for no real business reason.

thayne 2y ago
Or maybe it is price discrimination. A way to extract more money from customers that need higher availability and probably have higher budgets.
- hipadev23 2y ago
  Price discrimination is when you charge different amounts for the same thing to different customers. And usually the difference in those prices are not made apparent. Like when travel websites quote iOS users more than Android because they generally can afford to pay more. This is just regular ole pricing.
  - thayne 2y ago
    So what is the correct term for "charge an extremely high markup for a feature that some, but not all, of your customers need"?
    - spondylosaurus 2y ago
      Price gouging?
    - jchanimal 2y ago
      I came here to say the same thing. When you’re selling cloud services, the hardest thing to do is segment your customers by willingness to pay. Cross AZ traffic is exactly the sort of thing companies with budgets need, that small projects don’t.
    - mcmcmc 2y ago
      Supply and demand
  - hansvm 2y ago
    It's a bit of a mix, but price discrimination isn't far off. It's like the SSO tax; all organizations are paying for effectively the same service, but the provider has found a minor way to cripple the service that selectively targets people who can afford to pay more. If we want to call this just regular ole pricing, it's not a leap to call most textbook cases of price discrimination "regular ole pricing" as well. An online game charges more if your IP is from a certain geography? That's not discrimination; we've simply priced the product differently if you live in Silicon Valley; don't buy it it you don't want it.
    - hipadev23 2y ago
      https://en.wikipedia.org/wiki/Price_discrimination https://en.wikipedia.org/wiki/Price_discrimination Price discrimination has a clear definition. It’s not illegal in the US (when consumers are the victims anyway) but it has a clear meaning and you’re blurring the lines for I’m not sure what reason. Your example of a video game doing regional pricing is a perfect example of textbook price discrimination. Pricing a good or service at a level that inherently excludes those unwilling or unable to pay is: https://en.wikipedia.org/wiki/Excludability https://en.wikipedia.org/wiki/Excludability
      - hansvm 2y ago
        Alright, let's take a look at that first link as if it's gospel. AWS charging excessively for inter-AZ networking is: 1. a microeconomic pricing strategy 2. where largely similar goods (AWS with or without substantial inter-AZ bandwidth) 3. are sold at different prices (excessive inter-AZ networking fees) to different buyers 4. based on perceived market segments (most customers don't need (or don't know they need till they're locked in) much inter-AZ bandwidth, but larger, richer corporations likely do) I'm not trying to blur the lines. On top of any juggling of our favorite sources of definitions, that particular pricing strategy has all the qualitative hallmarks of price discrimination. Everyone still buys AWS, most customers are unaffected by the lack of bulk inter-AZ bandwidth, and AWS can successfully charge much more to those who can afford to pay.
      - thayne 2y ago
- cowsandmilk 2y ago
  Sending traffic between AZs doesn’t necessarily improve availability and can decrease it. Each of your services can be multi-az, but have hosts that talk just to other endpoints in their AZ.
  - thayne 2y ago
    Unless your app is completely stateless, you will need some level of communication across AZs. And you often want cross-zone routing on your load balancers so that if you lose all the instances in one AZ traffic will still get routed to healthy instances.
- KaiserPro 2y ago
  > Or maybe it is price discrimination. It very much is, because scaling bandwidth between phyical datacenters which are not located next to each other is very expensive. So pricing it means that people don't use it as much as if it was free.
  - mcmcmc 2y ago
    That’s not what price discrimination is
koolba 2y ago
You can still do that if you buffer it and push it to S3. Any AZ to S3 is free and the net result is the same.

KaiserPro 2y ago

I don't work for AWS However I do work for a company with >1 million servers. Scaling inter datacentre bandwidth is quite hard. Sure the datacentres might be geographically close, but laying network cables over distance is expensive. Moreover unless you spend uber millions, you're never going to get as much bandwidth as you have inside the datacentre. So you either apply hard limits per account, or price it so that people think twice about using it.

themgt 2y ago
OK but $10/TB has gotta be like >99% profit margin for AWS. After massively jacking up their prices, Hetzner internet egress is only €1/TB. Also AWS encourages / in some cases practically forces you to do multi-AZ. I remember switching to autoscaling spot instances to save a few bucks, then occasionally spot spinup would fail due to lack of availability within an AZ so I enabled multi-AZ spot. Then got hit with the inter-AZ bandwidth charges and wasn't actually saving any money vs single-AZ reserved. This was about the point I decided DIY Kubernetes was simpler to reason about.
- everfrustrated 2y ago
  Apples and Oranges. Hetzner doesn't even have multiple AZ's by AWS's definition - all Hetzners DCs eg Falkenstein 1-14 would be the same AZ zone. AWS network is designed with a lot more internal capacity and reliability than Hetzner which costs a lot more - multiple uplinks to independent switches, etc. AWS is also buying current gen network gear which is much more pricey - Hetzner is mostly doing 1 Gig ports or 10 gig at a push which means they can get away with >10 year old switches (if you think they buy new switches I have a bridge you might be interested in buying). This costs at least an order of magnitude more.
  - iscoelho 2y ago
    I agree with this post that Hetzner is a bad example. They are focused on a budget deployment. I do not agree that a state-of-the-art high capacity deployment is as expensive as you think it is. If an organization pays MSRP on everything, has awful procurement with nonexistent negotiation, and multiple project failures, sure, maybe. In the real world though, we're not all working for the federal government (-:

timewizard 2y ago

> It _should_ be cheap to operate HA systems across 2-3 AZs, In the steady state. HA systems tend towards large data bursts when failures or upgrades occur. > And at $0.01/GB, that's 2~10X what smaller providers charge for _internet_ egress. It's a lower latency network with a high SLA and automatic credits if the SLA isn't maintained. I think the inter-AZ option provides a level of service that's much higher than what most people want or need. It might be nice if there was a "best effort" inter-AZ network. This would probably fit better with the synchronization methods built into most HA software anyways. So, to me, it's a good product, it's just designed for a very niche segment of the market and often mistaken for something more general than it actually is.