6 ms·

The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gougi

The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response. With the huge margins they must have on egress bandwidth, I'm not holding my breath though.

datavirtue 5y ago
Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
- ianlevesque 5y ago
  You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
  - vbernat 5y ago
    To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
    - elithrar 5y ago
      The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
    - selcuka 5y ago
      Yet CloudFlare is able to do it for free up to a point, and at a lower-than-AWS fee after that. It might not be the exact figure but there is most definitely a huge markup there.
- gravypod 5y ago
  In the few times I've priced blended connections at most Colo locations unmetered 10G is ~500/month to ~1000/month. 5Gbps*1 month is ~1.5 PB. AWS is about 0.02/gb or ~30k/1.5pb. Approximately 30x the cost. These are old numbers for specific use cases so I'm not sure how much that has changed.
  - rstupek 5y ago
    AWS bandwidth is definitely over priced however the comparison to an unmetered colo isn't quite fair. What happens when that unmetered colo gets hit by a 100G ddos attack? Everything you have on it goes down?
    - mattzito 5y ago
      In fairness, Amazon also charges for more than basic DDoS protection: https://aws.amazon.com/shield/pricing/ https://aws.amazon.com/shield/pricing/ but regardless, the EGRESS charges are what are absurd. The only logical reason for charging so much for data exiting is to make sure that you can't practically leave the AWS ecosystem.
    - Dylan16807 5y ago
      If the discussion is about what it costs amazon, that's not really relevant. They avoid DDOS simply by being enormous.
  - jiggawatts 5y ago
    We have a customer on Azure running a bunch of medium-sized websites. Their mean outbound throughput is just 25 Mbps, but they're paying on the order of $3K/month all up for that Internet egress. (Not just bandwidth but some additional overpriced services on top like App Gateway.)
- GordonS 5y ago
  Nope, plenty of server hosts provide bandwidth for free, or near enough. Just one example is Hetzner, who include 20TB of bandwidth, with anything over that charged at just €1/TB. Meanwhile, AWS is gouging at €80/TB.
- ohazi 5y ago
  > Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? Come on, Amazon is not serving traffic through a Comcast business connection, they're peering directly with other large operators for free or for next to nothing.
  - DoctorOW 5y ago
    I mean that's another vector for savings... Maybe Cloudflare bought their own cable modem instead of paying the $20/month to rent one. :)
- piperswe 5y ago
  I have had to. I pay $400/month for the following: - A 42U cabinet - A 15A 120V circuit - An unmetered 1Gbps IP transit link All at a proper datacenter, namely Hurricane Electric's fmt2 facility. Includes a /29 of IPv4 and a /48 of IPv6, allows me to announce the /24 and /36 I own, and there's a free internet exchange onsite which I have a 1Gbps port at.
  - tatersolid 5y ago
    1800 Watts for a full rack? That’s bordering on useless. You can run maybe two real servers on 1800 W. You need to provision enough power for simultaneous startup after a power outage, unless you have some really smart PDUs and automation. We have a 10 year old DC with 30A@208V per rack and we have to leave racks half full because modern servers are so power-dense.
    - carlhjerpe 5y ago
      Gatekeeping server power!? We run 1U on 250-350w (230v). So yes, odd power allocation but servers are real way before 900w. We're running dual Xeon Gold low core count machines (because of Microsoft licensing) and they're pretty decent.
      - tatersolid 5y ago
        What’s the startup peak draw of those severs? Have you measured it? Remember everything turns on at once after an outage. In any case, even at your optimistic 250W per rack unit, a full rack of servers with two switches is more than 10kW!
        carlhjerpe 5y ago
        Absolutely, I'm with you. I just meant that 900w isn't where "real servers" start. If you're at the point where you're experiencing issues with scalable power on you could disable that feature and script ipmitool to power the servers on once ipmi is reachable but in a queue. Could help you eek out a bit more density. You could possibly run this fictional daemon on your management switch :)
- windexh8er 5y ago
  There are, literally, dozens of hosting providers that bake unlimited egress into their offerings - still. Bandwidth is cheap and has only gotten cheaper. Remember that Amazon doesn't always need to pay for Internet access. It's very likely they participate in Internet exchanges tied to other large providers. Given Amazon has a massive footprint they likely participate in these all over the globe to reduce congestion on their Internet connectivity. They very much have massive margin on egress. Given some of the cost comparisons floated today comparing R2 to S3 egress AWS is likely hitting 1000s of times (likely more) their return on the actual bandwidth cost they pay for month over month.
  - fomine3 5y ago
    Related: https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
  - sysihyk 5y ago
    CPU isn't cheap, typical PHP+MySQl pages are ridiculously heavy, that's why Cloudflare got themselves into business. Usually DoS attack doesn't exhaust the bandwidth, come on..
- seabrookmx 5y ago
  And your "business internet connection that is offered by ISPs" is a joke compared to the backbone AWS uses. Remember those ISPs need to make margin, that's why they charge what they do. AWS _could_ make it's margin on services alone. They just choose not to.
nindalf 5y ago
Like others have pointed out, it’s not about the margins on egress. Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly. It’s about creating a moat around all AWS services. If you’re already using some services inside the moat, you’re strongly encouraged to use the other services within the moat. If your data is in S3 or DynamoDB, egress fees will encourage you to process the data within AWS. You’re not going to use BigQuery for that. If you want to add a search index on that, you won’t go shopping around for search products on other clouds, you’ll likely use AWS elasticsearch. That’s where AWS makes money - through this soft lock-in. They’ll survive and maybe even thrive if they lose the revenue from egress. But who knows what’ll happen if the moat is destroyed? Every AWS service would need to compete on merit with every service on other clouds. They won’t be chosen simply by default.
- NicoJuicy 5y ago
  > Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly. Euh, 4-80x , the 80x is for EU and the US. https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/

datavirtue 5y ago

Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.

ianlevesque 5y ago
You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
- vbernat 5y ago
  To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
  - elithrar 5y ago
    The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
  - selcuka 5y ago
    Yet CloudFlare is able to do it for free up to a point, and at a lower-than-AWS fee after that. It might not be the exact figure but there is most definitely a huge markup there.
gravypod 5y ago
In the few times I've priced blended connections at most Colo locations unmetered 10G is ~500/month to ~1000/month. 5Gbps*1 month is ~1.5 PB. AWS is about 0.02/gb or ~30k/1.5pb. Approximately 30x the cost. These are old numbers for specific use cases so I'm not sure how much that has changed.
- rstupek 5y ago
  AWS bandwidth is definitely over priced however the comparison to an unmetered colo isn't quite fair. What happens when that unmetered colo gets hit by a 100G ddos attack? Everything you have on it goes down?
  - mattzito 5y ago
    In fairness, Amazon also charges for more than basic DDoS protection: https://aws.amazon.com/shield/pricing/ https://aws.amazon.com/shield/pricing/ but regardless, the EGRESS charges are what are absurd. The only logical reason for charging so much for data exiting is to make sure that you can't practically leave the AWS ecosystem.
  - Dylan16807 5y ago
    If the discussion is about what it costs amazon, that's not really relevant. They avoid DDOS simply by being enormous.
- jiggawatts 5y ago
  We have a customer on Azure running a bunch of medium-sized websites. Their mean outbound throughput is just 25 Mbps, but they're paying on the order of $3K/month all up for that Internet egress. (Not just bandwidth but some additional overpriced services on top like App Gateway.)
GordonS 5y ago
Nope, plenty of server hosts provide bandwidth for free, or near enough. Just one example is Hetzner, who include 20TB of bandwidth, with anything over that charged at just €1/TB. Meanwhile, AWS is gouging at €80/TB.
ohazi 5y ago
> Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? Come on, Amazon is not serving traffic through a Comcast business connection, they're peering directly with other large operators for free or for next to nothing.
- DoctorOW 5y ago
  I mean that's another vector for savings... Maybe Cloudflare bought their own cable modem instead of paying the $20/month to rent one. :)
piperswe 5y ago
I have had to. I pay $400/month for the following: - A 42U cabinet - A 15A 120V circuit - An unmetered 1Gbps IP transit link All at a proper datacenter, namely Hurricane Electric's fmt2 facility. Includes a /29 of IPv4 and a /48 of IPv6, allows me to announce the /24 and /36 I own, and there's a free internet exchange onsite which I have a 1Gbps port at.
- tatersolid 5y ago
  1800 Watts for a full rack? That’s bordering on useless. You can run maybe two real servers on 1800 W. You need to provision enough power for simultaneous startup after a power outage, unless you have some really smart PDUs and automation. We have a 10 year old DC with 30A@208V per rack and we have to leave racks half full because modern servers are so power-dense.
  - carlhjerpe 5y ago
    Gatekeeping server power!? We run 1U on 250-350w (230v). So yes, odd power allocation but servers are real way before 900w. We're running dual Xeon Gold low core count machines (because of Microsoft licensing) and they're pretty decent.
    - tatersolid 5y ago
      What’s the startup peak draw of those severs? Have you measured it? Remember everything turns on at once after an outage. In any case, even at your optimistic 250W per rack unit, a full rack of servers with two switches is more than 10kW!
      - carlhjerpe 5y ago
        Absolutely, I'm with you. I just meant that 900w isn't where "real servers" start. If you're at the point where you're experiencing issues with scalable power on you could disable that feature and script ipmitool to power the servers on once ipmi is reachable but in a queue. Could help you eek out a bit more density. You could possibly run this fictional daemon on your management switch :)
windexh8er 5y ago
There are, literally, dozens of hosting providers that bake unlimited egress into their offerings - still. Bandwidth is cheap and has only gotten cheaper. Remember that Amazon doesn't always need to pay for Internet access. It's very likely they participate in Internet exchanges tied to other large providers. Given Amazon has a massive footprint they likely participate in these all over the globe to reduce congestion on their Internet connectivity. They very much have massive margin on egress. Given some of the cost comparisons floated today comparing R2 to S3 egress AWS is likely hitting 1000s of times (likely more) their return on the actual bandwidth cost they pay for month over month.
- fomine3 5y ago
  Related: https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
- sysihyk 5y ago
  CPU isn't cheap, typical PHP+MySQl pages are ridiculously heavy, that's why Cloudflare got themselves into business. Usually DoS attack doesn't exhaust the bandwidth, come on..
seabrookmx 5y ago
And your "business internet connection that is offered by ISPs" is a joke compared to the backbone AWS uses. Remember those ISPs need to make margin, that's why they charge what they do. AWS _could_ make it's margin on services alone. They just choose not to.

nindalf 5y ago

Like others have pointed out, it’s not about the margins on egress. Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly. It’s about creating a moat around all AWS services. If you’re already using some services inside the moat, you’re strongly encouraged to use the other services within the moat. If your data is in S3 or DynamoDB, egress fees will encourage you to process the data within AWS. You’re not going to use BigQuery for that. If you want to add a search index on that, you won’t go shopping around for search products on other clouds, you’ll likely use AWS elasticsearch. That’s where AWS makes money - through this soft lock-in. They’ll survive and maybe even thrive if they lose the revenue from egress. But who knows what’ll happen if the moat is destroyed? Every AWS service would need to compete on merit with every service on other clouds. They won’t be chosen simply by default.

NicoJuicy 5y ago
> Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly. Euh, 4-80x , the 80x is for EU and the US. https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/