4 ms·

I don't work for AWS However I do work for a company with >1 million servers. Scaling inter datacentre bandwidth is quite hard. Sure the datacentres might be g

I don't work for AWS However I do work for a company with >1 million servers. Scaling inter datacentre bandwidth is quite hard. Sure the datacentres might be geographically close, but laying network cables over distance is expensive. Moreover unless you spend uber millions, you're never going to get as much bandwidth as you have inside the datacentre. So you either apply hard limits per account, or price it so that people think twice about using it.

themgt 2y ago
OK but $10/TB has gotta be like >99% profit margin for AWS. After massively jacking up their prices, Hetzner internet egress is only €1/TB. Also AWS encourages / in some cases practically forces you to do multi-AZ. I remember switching to autoscaling spot instances to save a few bucks, then occasionally spot spinup would fail due to lack of availability within an AZ so I enabled multi-AZ spot. Then got hit with the inter-AZ bandwidth charges and wasn't actually saving any money vs single-AZ reserved. This was about the point I decided DIY Kubernetes was simpler to reason about.
- everfrustrated 2y ago
  Apples and Oranges. Hetzner doesn't even have multiple AZ's by AWS's definition - all Hetzners DCs eg Falkenstein 1-14 would be the same AZ zone. AWS network is designed with a lot more internal capacity and reliability than Hetzner which costs a lot more - multiple uplinks to independent switches, etc. AWS is also buying current gen network gear which is much more pricey - Hetzner is mostly doing 1 Gig ports or 10 gig at a push which means they can get away with >10 year old switches (if you think they buy new switches I have a bridge you might be interested in buying). This costs at least an order of magnitude more.
  - iscoelho 2y ago
    I agree with this post that Hetzner is a bad example. They are focused on a budget deployment. I do not agree that a state-of-the-art high capacity deployment is as expensive as you think it is. If an organization pays MSRP on everything, has awful procurement with nonexistent negotiation, and multiple project failures, sure, maybe. In the real world though, we're not all working for the federal government (-:
  - dantillberg 2y ago
    While your caveats are all noteworthy, I'll add that Hetzner also offers unlimited/free bandwidth between their datacenters in Germany and Finland. That's sort of like AWS offering free data transfer between us-east-1 and us-east-2.
iscoelho 2y ago
In Ashburn, VA, I can buy Dark Fiber for $750 MRC to any datacenter in the same city. I can buy Dark Fiber for $3-5K MRC to any random building in the same city. That Duplex Dark Fiber with DWDM can run 4TBPS of capacity at 100GE (40x 100GE). Each 100GE transceiver costs $2-4K NRC dependent on manufacturer - $160K NRC for 40x. (There are higher densities as well, like 200/400/800GE, 100GE is just getting cheap.) In AWS, utilizing 1x100GE will cost you >$1MM MRC. For significantly less than that, let's say absolutely worst-case 5K MRC + 200K NRC, you can get 40x100GE. Now you have extra money for 4x redundancy, fancy routers, over-spec'd servers, world-class talent, and maybe a yacht if your heart desires.
- bobbob1921 2y ago
  I’m just throwing out a hypothetical, so I may be completely off base: perhaps aws charges high inter-AZ bandwidth prices to keep users from tunneling traffic between availabilities zones to arbitrage lower Internet/egress costs at AZ 1 vs AZ 3. Outside of my statement above, I do agree that the cost Amazon pays for bandwidth between their sites, has to be practically nothing at their scale/size (and thus they should charge their customers very little for it, especially considering easy-multi AZ is a big differentiator for cloud vs self-hosting / colo). The user above’s dark fiber MRC prices are spot on.
  - paranoidrobot 2y ago
    > availabilities zones to arbitrage lower Internet/egress costs at AZ 1 vs AZ 3. Unless I've missed something - They charge the same price for all AZs in a given region. The only exception for any AWS service that I'm aware of, is EC2 Spot instance pricing.
dilyevsky 2y ago
You should do the math though because it’s expensive but nowhere near 1c/g expensive
znpy 2y ago
Amazon does have uber millions though
rendaw 2y ago
That doesn't square with what GP wrote about S3 bandwidth costs. And IIUC GP's point is that it's priced much above cost.

themgt 2y ago

OK but $10/TB has gotta be like >99% profit margin for AWS. After massively jacking up their prices, Hetzner internet egress is only €1/TB. Also AWS encourages / in some cases practically forces you to do multi-AZ. I remember switching to autoscaling spot instances to save a few bucks, then occasionally spot spinup would fail due to lack of availability within an AZ so I enabled multi-AZ spot. Then got hit with the inter-AZ bandwidth charges and wasn't actually saving any money vs single-AZ reserved. This was about the point I decided DIY Kubernetes was simpler to reason about.

everfrustrated 2y ago
Apples and Oranges. Hetzner doesn't even have multiple AZ's by AWS's definition - all Hetzners DCs eg Falkenstein 1-14 would be the same AZ zone. AWS network is designed with a lot more internal capacity and reliability than Hetzner which costs a lot more - multiple uplinks to independent switches, etc. AWS is also buying current gen network gear which is much more pricey - Hetzner is mostly doing 1 Gig ports or 10 gig at a push which means they can get away with >10 year old switches (if you think they buy new switches I have a bridge you might be interested in buying). This costs at least an order of magnitude more.
- iscoelho 2y ago
  I agree with this post that Hetzner is a bad example. They are focused on a budget deployment. I do not agree that a state-of-the-art high capacity deployment is as expensive as you think it is. If an organization pays MSRP on everything, has awful procurement with nonexistent negotiation, and multiple project failures, sure, maybe. In the real world though, we're not all working for the federal government (-:
- dantillberg 2y ago
  While your caveats are all noteworthy, I'll add that Hetzner also offers unlimited/free bandwidth between their datacenters in Germany and Finland. That's sort of like AWS offering free data transfer between us-east-1 and us-east-2.

iscoelho 2y ago

In Ashburn, VA, I can buy Dark Fiber for $750 MRC to any datacenter in the same city. I can buy Dark Fiber for $3-5K MRC to any random building in the same city. That Duplex Dark Fiber with DWDM can run 4TBPS of capacity at 100GE (40x 100GE). Each 100GE transceiver costs $2-4K NRC dependent on manufacturer - $160K NRC for 40x. (There are higher densities as well, like 200/400/800GE, 100GE is just getting cheap.) In AWS, utilizing 1x100GE will cost you >$1MM MRC. For significantly less than that, let's say absolutely worst-case 5K MRC + 200K NRC, you can get 40x100GE. Now you have extra money for 4x redundancy, fancy routers, over-spec'd servers, world-class talent, and maybe a yacht if your heart desires.

bobbob1921 2y ago
I’m just throwing out a hypothetical, so I may be completely off base: perhaps aws charges high inter-AZ bandwidth prices to keep users from tunneling traffic between availabilities zones to arbitrage lower Internet/egress costs at AZ 1 vs AZ 3. Outside of my statement above, I do agree that the cost Amazon pays for bandwidth between their sites, has to be practically nothing at their scale/size (and thus they should charge their customers very little for it, especially considering easy-multi AZ is a big differentiator for cloud vs self-hosting / colo). The user above’s dark fiber MRC prices are spot on.
- paranoidrobot 2y ago
  > availabilities zones to arbitrage lower Internet/egress costs at AZ 1 vs AZ 3. Unless I've missed something - They charge the same price for all AZs in a given region. The only exception for any AWS service that I'm aware of, is EC2 Spot instance pricing.

dilyevsky 2y ago

You should do the math though because it’s expensive but nowhere near 1c/g expensive

znpy 2y ago

Amazon does have uber millions though

rendaw 2y ago

That doesn't square with what GP wrote about S3 bandwidth costs. And IIUC GP's point is that it's priced much above cost.