4 ms·

The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of

The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of all of the magic that happens inside AWS between those two things. I've seen some of the inner workings of the big cloud providers' networking stacks. The networking infrastructure, the software that runs it, the software that exposes it to customers, the thousands of engineers working at any given moment in AWS/GCP/Azure's NOCs to maintain uptime are truly some of the most impressive technical marvels I have ever seen. They aren't as sexy to discuss on HN as something like the managed containers services, functions as a service, EC2 etc, but the networking stacks like the VPC, NAT gateways, subnet routing, privatelinks, security groups, ENIs, nitro cards, etc are pure magic as far as I'm concerned and are so so so much more complicated than a standard data center's networking stack, or even Cloudflare's stack. To use Cloudflare's "bucket of water" metaphor, AWS isn't even close to just being a dumb bucket of water that you fill with water and then get charged to take out the water. There is so much that happens inside of that bucket to segment your water into different pipes, routing your water in all kinds of customer-customizable ways for many different use cases, mixing/heating/cooling your water as you need, all while guaranteeing things like making sure your water arrives exactly where it is supposed to arrive and doesn't get contaminated or leaked along the way. Does AWS make a big markup on bandwidth? Yea, surely they do. But is it as simple as Cloudflare says it is? Not even close.

VectorLock 5y ago
It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.
- runlevel1 5y ago
  S3 has its own data retrieval costs, as do several of their managed services. Those are separate charges from the EC2 costs the Cloudflare blog post discussed.
- Guvante 5y ago
  Egress does not mean S3 or Cloudflare. Egress is the raw pipe billing from AWS to the wider internet. Other services are priced differently.
  - onphonenow 5y ago
    That’s where folks are revealing how clueless they are. Raw pipe isn’t priced in GB it’s in capacity. To serve GB on Black Friday you need to provision far far more capacity
    - Dylan16807 5y ago
      Yeah but people can do that math. Peak bandwidth is not 200x higher than average bandwidth. Cloudflare used a 5x multiplier. How high do you think it needs to be? Does total AWS bandwidth even go up that much on black friday?
starfallg 5y ago
It's not really that much magic. It's just a variation of EVPN-VXLAN plus smart NICs that segments and directs the traffic. Then they have normal VM hosts or nowadays devices with ASICs that handle the GW and NAT functionality.
- sofixa 5y ago
  Custom ASICs (Nitro chips) aren't magic? Maybe so, but they cost money to develop. All of the other networking stuff ( Security Groups, NACLs, flow logs, VPCs, subnets, etc.) you don't directly pay for, isn't magic either, but also cost money.
  - deleted 5y ago
    [deleted]
  - starfallg 5y ago
    Nitro is just a fancy converged host adapter with Smart NIC functionality. It's unclear to the industry how much of Nitro is custom, and how much of it is existing IP that is cobbled together (e.g. Graviton and the ARM Neoverse cores). The ASICs are on the fabric doing the routing and NAT for all the traffic in the AZ. These ASIC are unlikely to be custom. Hyperscale operators typically use open networking hardware with merchant silicon. You can get open networking hardware to do all sorts of packet manipulation, and these devices are a cheaper than traditional manufacturers, but more powerful as they expose more low-level interfaces. All those features you talk about are implemented from features that are provided by these hardware platforms. AWS is just putting an managed service together from them, no different to how they take postgres, do some tweaks and rebrand it as an AWS service.
lewisl9029 5y ago
Yes, their network stacks are definitely complex and cost a lot to maintain, I'm sure. But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. In fact it makes the offering less attractive if the complexity isn't sufficiently abstracted away and distract from product work or if their abstractions are leaky. Recently I've been working with https://fly.io/ https://fly.io/ for a new app and it's a breath of fresh air compared to working with the big cloud providers. They offer simple but robust networking primitives built on top of ipv6 and WireGuard and provide a ton of value add on top like global distribution & load balancing, service discovery, TLS termination, all of which just work exactly like I'd expect it to, out of the box without any configuration on my side. EDIT: Almost forgot to mention: their egress costs are also much more reasonable: https://fly.io/docs/about/pricing/#outbound-data-transfer https://fly.io/docs/about/pricing/#outbound-data-transfer
- nyokodo 5y ago
  > But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. It’s simply obvious that it’s not a good deal if you’re not their target customer with a use case they cater to. However, it could be a good deal if you have a relevant use case. Unless it’s being suggested that AWS caters to everyone in all cases then it adds nothing to the conversation to point it out.
- podc 5y ago
  I'm watching fly.io with interest, I want to see how they handle the first major incidents - response time, lessons learnt, transparency before I trust them with a production site though. Most SRE skills related to your own operations are all learnt on the battlefield and not via some cliche must-read book from Google engineers afaic. If its Linode style - delayed status page updates - sometimes as much as 15minutes, zero detail post-mortems - this problem has been fixed by our engineers thank you yada yada, and same issues repeat six months down the line then I will be understandably disappointed.
  - lewisl9029 5y ago
deleted 5y ago
[deleted]
mythz 5y ago
https://www.hetzner.com/cloud https://www.hetzner.com/cloud gives you 20TB bandwidth for €3.49/mo VMs, which I've essentially regarded as Hetzner gives unlimited free bandwidth for all servers. Being lynched for egregious egress fees is only something I've experienced when using mega corp's clouds, where economies of scale suggests their vastly larger size should allow them to provide even better value. But that's in a normal market, not the artificial lock-in mega cloud corps enjoy where they're able to distort customer behavior from artificially high pricing.
- stillicidious 5y ago
  I'm a Hetzner home user and a huge fan, but let's not compare the quality of networking you get for free from them with the networking you get from AWS. I don't think I've seen a latency spike on AWS in 10 years. Hetzner, it's often possible to observe latency and drops over 10 minutes (and the situation hasn't changed much in about 10 years)
  - mythz 5y ago
    In all the years I've used Hetzner I've never observed these random 10 minute latency drops you speak of. They've always had much faster internet access then I've ever been able to get from my home broadband so I'll even SSH into & use them for network intensive dev tasks like iterating on a new Docker container since it's able to download & build the image packages in a fraction of the time. The primary issue I have with them is latency access to their DE/FI data centers from the US, if their US DC offered dedicated servers I would be migrating to over to use them instead.
    - stillicidious 5y ago
      They launched Cloud in the US this month, very likely dedicated will be offered soon enough. The bang for buck on Hetzner is insane, really love them, but have and would rip them out of any business environment I come across, largely due to network quality and attitude to support. If you haven't experienced Google translating insistently German responses from one of their DC techs you probably haven't been using them for long enough ;) As for networking, would encourage installing something like Smokeping

VectorLock 5y ago

It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.

runlevel1 5y ago
S3 has its own data retrieval costs, as do several of their managed services. Those are separate charges from the EC2 costs the Cloudflare blog post discussed.
Guvante 5y ago
Egress does not mean S3 or Cloudflare. Egress is the raw pipe billing from AWS to the wider internet. Other services are priced differently.
- onphonenow 5y ago
  That’s where folks are revealing how clueless they are. Raw pipe isn’t priced in GB it’s in capacity. To serve GB on Black Friday you need to provision far far more capacity
  - Dylan16807 5y ago
    Yeah but people can do that math. Peak bandwidth is not 200x higher than average bandwidth. Cloudflare used a 5x multiplier. How high do you think it needs to be? Does total AWS bandwidth even go up that much on black friday?

starfallg 5y ago

It's not really that much magic. It's just a variation of EVPN-VXLAN plus smart NICs that segments and directs the traffic. Then they have normal VM hosts or nowadays devices with ASICs that handle the GW and NAT functionality.

sofixa 5y ago
Custom ASICs (Nitro chips) aren't magic? Maybe so, but they cost money to develop. All of the other networking stuff ( Security Groups, NACLs, flow logs, VPCs, subnets, etc.) you don't directly pay for, isn't magic either, but also cost money.
- deleted 5y ago
  [deleted]
- starfallg 5y ago
  Nitro is just a fancy converged host adapter with Smart NIC functionality. It's unclear to the industry how much of Nitro is custom, and how much of it is existing IP that is cobbled together (e.g. Graviton and the ARM Neoverse cores). The ASICs are on the fabric doing the routing and NAT for all the traffic in the AZ. These ASIC are unlikely to be custom. Hyperscale operators typically use open networking hardware with merchant silicon. You can get open networking hardware to do all sorts of packet manipulation, and these devices are a cheaper than traditional manufacturers, but more powerful as they expose more low-level interfaces. All those features you talk about are implemented from features that are provided by these hardware platforms. AWS is just putting an managed service together from them, no different to how they take postgres, do some tweaks and rebrand it as an AWS service.

lewisl9029 5y ago

Yes, their network stacks are definitely complex and cost a lot to maintain, I'm sure. But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. In fact it makes the offering less attractive if the complexity isn't sufficiently abstracted away and distract from product work or if their abstractions are leaky. Recently I've been working with https://fly.io/ https://fly.io/ for a new app and it's a breath of fresh air compared to working with the big cloud providers. They offer simple but robust networking primitives built on top of ipv6 and WireGuard and provide a ton of value add on top like global distribution & load balancing, service discovery, TLS termination, all of which just work exactly like I'd expect it to, out of the box without any configuration on my side. EDIT: Almost forgot to mention: their egress costs are also much more reasonable: https://fly.io/docs/about/pricing/#outbound-data-transfer https://fly.io/docs/about/pricing/#outbound-data-transfer

nyokodo 5y ago
> But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. It’s simply obvious that it’s not a good deal if you’re not their target customer with a use case they cater to. However, it could be a good deal if you have a relevant use case. Unless it’s being suggested that AWS caters to everyone in all cases then it adds nothing to the conversation to point it out.
podc 5y ago
I'm watching fly.io with interest, I want to see how they handle the first major incidents - response time, lessons learnt, transparency before I trust them with a production site though. Most SRE skills related to your own operations are all learnt on the battlefield and not via some cliche must-read book from Google engineers afaic. If its Linode style - delayed status page updates - sometimes as much as 15minutes, zero detail post-mortems - this problem has been fixed by our engineers thank you yada yada, and same issues repeat six months down the line then I will be understandably disappointed.
- lewisl9029 5y ago

deleted 5y ago

[deleted]

mythz 5y ago

https://www.hetzner.com/cloud https://www.hetzner.com/cloud gives you 20TB bandwidth for €3.49/mo VMs, which I've essentially regarded as Hetzner gives unlimited free bandwidth for all servers. Being lynched for egregious egress fees is only something I've experienced when using mega corp's clouds, where economies of scale suggests their vastly larger size should allow them to provide even better value. But that's in a normal market, not the artificial lock-in mega cloud corps enjoy where they're able to distort customer behavior from artificially high pricing.

stillicidious 5y ago
I'm a Hetzner home user and a huge fan, but let's not compare the quality of networking you get for free from them with the networking you get from AWS. I don't think I've seen a latency spike on AWS in 10 years. Hetzner, it's often possible to observe latency and drops over 10 minutes (and the situation hasn't changed much in about 10 years)
- mythz 5y ago
  In all the years I've used Hetzner I've never observed these random 10 minute latency drops you speak of. They've always had much faster internet access then I've ever been able to get from my home broadband so I'll even SSH into & use them for network intensive dev tasks like iterating on a new Docker container since it's able to download & build the image packages in a fraction of the time. The primary issue I have with them is latency access to their DE/FI data centers from the US, if their US DC offered dedicated servers I would be migrating to over to use them instead.
  - stillicidious 5y ago
    They launched Cloud in the US this month, very likely dedicated will be offered soon enough. The bang for buck on Hetzner is insane, really love them, but have and would rip them out of any business environment I come across, largely due to network quality and attitude to support. If you haven't experienced Google translating insistently German responses from one of their DC techs you probably haven't been using them for long enough ;) As for networking, would encourage installing something like Smokeping