17 ms·

AWS's Egregious Egress

742 points by jgrahamc 5y ago

PaulHoule 5y ago
Does this make sense? He seems to be comparing peak bandwidth pricing (megabits per second) to average throughput pricing (megabits). Peak bandwidth pricing is really a lie (if you actually used it all the time you'd be looking for a new host)
- eastdakota 5y ago
  Calculated assuming 20% utilization, which is the low end of industry average (20% – 40%).
cameronh90 5y ago
"In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher. We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn't seem to be an innocent explanation. " One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that?
- sitkack 5y ago
  > One consideration could be that you can't directly control data ingress. Are you thinking UDP? Or for a DDOS? Connection setup overhead could be accounted for. I highly doubt this is the reason.
  - linuxftw 5y ago
    If there's an IP exposed on the internet, you can just send it tcp payloads. The end destination will silently drop them, but it doesn't mean people can't send you gigs of useless data.
    - otterley 5y ago
      I think you mean UDP. TCP requires a 3-way handshake first.
      - ithkuil 5y ago
        Intermediate routers don't care about that; they only forward the IP packets; four target host/firewall will drop them (because they don't belong to a valid connection) but they will be still accounted for as ingress traffic.
        otterley 5y ago
        Correct, but you can't get much bandwidth through until the 3-way handshake is completed. Sending a bunch of unanswered SYN packets isn't really a great way to instigate a DDoS, compared to sending avalanches of 64KiB UDP packets.
        ithkuil 5y ago
        As long as there is no connection tracking you can send whatever crap you want, including replayed packets from the middle of a connection, perhaps even huge packets with a syn flag ... As long as the accounting happens before a firewall performs basic TCP sanity checks you're going to pay for it
        kazen44 5y ago
        in some setups, you could send non IP packets (especially in MPLS fabric if accounting happens by LSP).
  - blamethenetwork 5y ago
    At a routing and peering level. Once you have an announcement for your netblock out there, traffic will start to head towards it. A lot of this is due to the BGP Path Selection Algorithm. You can try and influence how traffic arrives, by doing things like, AS prepends, but you are still going to get traffic. The main reason for this is that the other side that is egressing to you has their own egress policy that also follows path selection. Things like localpref and weight will force my traffic to leave via a path before it considers how a network has AS padded. As an example: Lets say I want to egress (company A) to a downstream company (company B). If I learn routes to Company B via multiple ways: peering fabric (low cost), paid peering (medium cost), transit1 (high cost, variable quality), transit2 (low cost variable quality), I can choose which way my traffic goes, via localpref, weight etc. Only when I view the paths equally (equal localpref, weight etc.) will I evaluate the shortest AS Path (which the receiving company has influence on). The only way to completely not get inbound traffic via a specific link, is to remove your BGP advertisement for your netblock from that link. (some providers also let you do this selectively via BGP communities). There are also some other tips/tricks - such as adding a more specific prefix to a certain link, to attract traffic, but care needs to be made to have a fallback route in case things go wonky.
- tw04 5y ago
  Coming from someone who believes that the AWS egress fees are highway robbery, I'd throw this out there. In order to maintain settlement free peering, you need to keep egress and ingress somewhat even with your peers. Amazon proper probably has 3x egress vs ingress between prime and twitch. AWS is an easy avenue to balance it out. All that being said: I would guess there's almost 0 chance that's what's happening here. Amazon is just using it as a form of lock-in IMO.
  - hnpasf 5y ago
    I agree that it is just lock-in. The price of egress is the same even when using devices like the Snowball where the data is loaded onto a server and physically shipped to you.
- qeternity 5y ago
  > One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. Most request/response models are very asymmetric. I can request a relatively large asset from S3 with a small request and maliciously generate fees as an attacker. AWS egress is vendor lock-in 100%.
- eli 5y ago
  It's also just a way to segment customers. The people doing tons of egress, on average, probably have bigger budgets for cloud computing. There's no particular reason AWS costs have to mirror how they charge.
- res0nat0r 5y ago
  The only service AWS hasn't consistently lowered pricing on (unlike almost everything else) is their egress cost. Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board. But egress is the exception. They're keeping pricing high for this exact reason to prevent folks from moving their data elsewhere. I've got about 25TB of stuff I'd love to keep all in Glacier Deep Archive, but the restore costs are just too insane to justify due to the egress pricing still at .09c after all these years. Too bad.
  - skuhn 5y ago
    I view network egress as the primary lock-in mechanism for all of AWS. If you want to migrate to another provider, network egress costs mean that you'll spend multiples of your normal monthly operating costs to do so. That stifles competition. But well before that point, most services in AWS have already been built around avoiding network egress wherever possible. You're always going to prefer AWS APIs and services, even if they aren't the best for your use case, because services outside AWS have a network egress tariff placed on them. So you aren't always buying best-of-breed, you're buying the best of what AWS chooses to offer (or the selection of vendors that choose to build in AWS to remain competitive). And if that isn't good enough, your only other option is to migrate out (and pay those egress costs!).
  - qeternity 5y ago
    > Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board. How would keeping prices high for these services keep people locked in? The whole point of vendor lock in is there is some other indirect cost that you bear by switching, which is why you don't switch. For AWS, this indirect cost is bandwidth. It is the lock in mechanism.
    - res0nat0r 5y ago
      > How would keeping prices high for these services keep people locked in? By doing the opposite and raising prices across the board randomly because they know folks can't spend the time and money to migrate without a large support burden or time/money sunk cost. The only service not going down in price is egress and the only thing I've seen which has been like this, to directly discourage folks from migrating their data to say GCP when their offering(s) look more attractive.
- isbvhodnvemrwvn 5y ago
  If you get DDoSed you also pay through the roof in fees (for whatever is in your VPC - e.g. ELB which are billed per throughput) unless you buy their anti-DDoS services (AWS Shield Advanced) for 3k$/mo - without that AWS won't waive the fees.
- RKearney 5y ago
  > One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that? An equally likely reason is by making ingress free it helps balance out ingress and egress traffic flows which are usually desirable for settlement-free peering agreements.
- kevincox 5y ago
  The simple explanation is that ingress is far, far lower than egress so the cost isn't relevant since links tend to be symmetrical. So the cost to them is determined based on egress, their ingress provisioning happens at no additional cost.
thejosh 5y ago
Azure discount for Cloudflare seems tiny though, but that won’t fit into their recent marketing efforts Cloudflare has been doing.
- pumanoir 5y ago
  I didn't see the amount of the discount in the article. Is there a place where one can see the pricing (i.e. 10TB azure -> CF)?
  - donmcronald 5y ago
    I couldn’t find it when I searched. Some BA participants have fair usage limits too. Ex: I think Wasabi is about 1x your stored data per month.
kbuck 5y ago
I've assumed this was well-known... AWS has never been a good deal on the billing side. You're paying a huge premium for API access and the strength of the AWS brand. I was a bit confused as to why Cloudflare was blogging about it until halfway through the article, when The Bandwidth Alliance came in. Makes sense now: they just want to shame Amazon into offering free egress bandwidth to Cloudflare. This is basically a hit piece. Also, another aside: for hosting providers (AWS included), ingress bandwidth is typically free because it's also basically free for the provider. Outbound dominates so much that the inbound traffic is a rounding error if it's billed by their transit providers at all. But I agree that the high egress prices are a vendor lock-in strategy.
- deleted 5y ago
  [deleted]
- pkilgore 5y ago
  Perhaps I misunderstood, but the article says Amazon already does not pay for egress to Cloudflare. But Amazon still charges their customers as if they did. Did I misunderstand something?
  - inopinatus 5y ago
    You sure did. Peering isn’t free, or even cheap. Cloudflare are whining for special treatment for their bandwidth cartel, and AWS are large enough to tell them to get fucked. The key is realising that Cloudflare have always been about disingenuously leveraging other people’s stuff, since they’re basically just a middleman.
    - Dylan16807 5y ago
      Peering's definitely cheap compared to what Amazon's charging. Amazon even has a peering product for 16.5 cents per Mbps, but it's not usable in this circumstance. "Please join our free peering group of many companies." is neither a cartel nor a request for special treatment. What are they "disingenuously leveraging"?
edoceo 5y ago
Reads like CF has a bone to pick with AWS. Are they mad about AWS not in the peering group? Is the AWS price really out of line - the comparison to wholesale cost only seems to miss a lot of infrastructure that gets baked into the final retail prices - it's not simply (Retail-Wholesale=7000% margin). Feels like a weak "hit piece" to me.
- notsureaboutpg 5y ago
  Cloudflare is ramping up it's enterprise sales and deals (and doing a good job of it too, anecdotally). So they are competing more and more with AWS (their Workers product is a good example), and why not point out things people dislike about their competitors?
- Daishiman 5y ago
  I think it's quite fair. You're not paying anywhere near close to the price of the infra with AWS; you're paying for the convenience, but be mindful that the cost has a break-even point and many companies may be beyond that point. I don't have the experience to assess whether the total cost of ownership is favorable to most AWS clients or not.
- pkilgore 5y ago
  They directly address these costs in the article as "rounding error". Do you have some evidence to dispute that? Did I misunderstand them?
- isbvhodnvemrwvn 5y ago
  They are in the market for reducing that bandwidth by caching stuff at CloudFlare, so it's not exactly surprising.
AgentK20 5y ago
Working in realtime games that require high bandwidth usage, AWS and basically every other public cloud is fundamentally unusable for us because of this exact problem. Our bare metal infrastructure for the Hypixel Minecraft network uses 3-4PB per month, so we simply lease a 100gbps transit link billed at 95th percentile. Last I checked, AWS wanted ~$200k per MONTH, for just bandwidth (no compute, memory, storage, or anything else). We'd love to be able to use the cloud, but not at the cost of increasing our monthly expenses by an entire order of magnitude, so we just stick to bare metal colocation. I honestly believe that if you have the technical skill in-house, and are spending more than, say, $30k/mo on public cloud hosting, that you should seriously evaluate whether bare metal could significantly decrease your costs.
- aclelland 5y ago
  At $JOB, we have a similar issue with static content. We serve over 1PB a month and the price AWS would charge to use CloudFront is an order of magnitude larger than even a Cloudflare Enterprise plan (which comes with some nice bells and whistles that Cloudfront doesn't offer). Even with discounts from AWS it just doesn't make sense to use AWS to serve up the assets to users. We do use S3 as our static asset backend and the combination works really well. I would love to see Cloudflare release a S3 compatible storage service though. I think we'd jump onto that in a heartbeat.
  - HeavenFox 5y ago
    Backblaze has S3-compatible API and they have free data transfer to Cloudflare
    - aclelland 5y ago
      The last time I looked at BB S3 API they didn't offer the lifecycle controls that AWS over. Mainly the ability to remove old versions of files after a period of time and we didn't want to roll our own expiry solution. Might be worth looking again though.
      - 5y ago
api 5y ago
It's very much by design. They want ingress not egress to create "gravity." The end game is for virtually all compute to happen in the cloud and be accessed only by thin client. This pricing drives things in this direction by making companies that would use the cloud to just service local compute less profitable than companies that host everything in the cloud and provide only web or other thin client access. There are providers that don't do this, but they are much less batteries-included. The best bandwidth deals are via bare metal hosting which is basically just box-in-a-rack rental. You can get bandwidth there by capacity rather than transfer quota. It's many many orders of magnitude cheaper.
derefr 5y ago
> The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first. Presuming the average AWS customer is running an API SaaS, said customer’s API response load (= egress bandwidth) will tend to be a good measure of how much “useful work” said SaaS is doing for its own clients — either in the form of API responses, requests to third parties, emails sent, notifications pushed, uploads to other providers, etc. As such, egress bandwidth tends to be a good measure of having product-market fit, and thereby of willingness-to-pay. Things that trigger mostly ingress load — dropping attacks at a firewall, accepting uploads, receiving emails/notifications, etc. aren’t as emblematic of product-market fit (they aren’t things you would traditionally charge customers for “by volume”), but rather are usually activities done as loss-leaders; and so do not correlate to willingness-to-pay.
alvis 5y ago
Back in the old day, we were charged by the maximum bandwidth. Not quite scalable maybe, but it definitely won’t bankrupt you if you’re not careful.
mdasen 5y ago
On the Hotel California Effect If it's expensive to move data out of AWS, it's not just about making it hard for customers to leave AWS. It means that any third-party service that wants to sell to AWS customers must also use AWS. For example, Snowflake can't really run its own data centers. They need to rent from AWS. If you're a company on AWS, you won't want to spend $0.08/GB sending things to Snowflake. The high egress cost gives Amazon a lot of power to keep the third-party ecosystem within AWS and without as much negotiating power on things like EC2 costs. Snowflake can't say to Amazon, "if you don't give us a great discount, we're going to move our operations to DigitalOcean" because Amazon would just say, "no you're not. Your customers aren't going to pay $80/TB to load things into Snowflake when you're charging them $23-40/TB for storage. They'd be paying 2-3.5x your storage costs just to load the data into your system!" Yes, I'm sure that Snowflake does get a discount from AWS, but having less negotiating power can make discounts smaller. There are other reasons to want a third party to use AWS if you're on AWS. Still, the egress pricing seems to make it very hard for third-party tech providers not to use AWS if their customers are using it.
- Hokusai 5y ago
  Amazon is really good at leveraging its semi monopolistic presence in different markets. Once you are big enough you only need to tweak a few parameters to dominate the market. It's interesting how AWS uses egress to this effect. Your comment is very insightful.
  - stadium 5y ago
    They employ teams of economists to model price elasticity of demand. There are likely many orders of magnitude more than a few parameters. And they look at how a change in one product feature or pricing model cannibalizes or compliments other features and prices.
- nindalf 5y ago
  Thank you for this. I’m surprised that CloudFlare didn’t mention it in an otherwise thorough post. After all, the reason they’re trying to pressure AWS to change is that they’d like to offer services to AWS customers without bandwidth markup. Essentially competing with existing AWS services on an even footing. This to me makes it unlikely that AWS will change their bandwidth pricing. If it was just about this revenue source disappearing, they might bite the bullet and hope that the lower prices would attract more customers in the long term. But bandwidth pricing is the moat around their castle. If it didn’t exist, Snowflake and others would leave. But more than that, new AWS services benefit from the captive audience choosing them by default. Without the moat, each service has to compete on its merits. AWS Lambda, for example would need to be better than CloudFlare Workers or fly.io containers. It won’t win customers like it does today just because it’s “free” bandwidth. Removing these fees is a risk AWS is currently choosing not to take. And they won’t change their pricing regardless of the number of blog posts that competitors write. The only thing that would force a change is an exodus of customers leaving AWS citing this as a reason.
donmcronald 5y ago
It looks like Cloudflare cut egress pricing for workers unbound to $.0045 per GB, but that’s still half of AWS or “40x markup” for me in Canada. The original workers model of pay per execution is incredible. IIRC I figured one run is the equivalent of 6KB of egress at AWS. Is there a point where Cloudflare will start charging egress on those if I transfer too much? I want to use it for something, but don’t like the “free bandwidth until you hit the invisible limit” side of things Cloudflare has going. 1. https://blog.cloudflare.com/workers-unbound-ga/ https://blog.cloudflare.com/workers-unbound-ga/
- satyrnein 5y ago
  There's a typo, it's $0.045 per GB (half of AWS, as mentioned). I got excited for a second!
  - donmcronald 5y ago
    Damn. I'm stupid today. I don't think I can edit anymore either :-(
- deleted 5y ago
  [deleted]
- yepcfstyle 5y ago
  Cloudflare argo pricing (basically stay on the CF network) is almost identical to GCP Premium Tier and AWS default tier. They claim it does some cool optimizations to traffic path, no different to BGP optimizers - it is simply ridiculous to presume GCP/AWS doesnt optimize their routes too on a continous basis - yet apples and oranges doesn't apply CF says. I applaud CF nudging AWS to reduce bandwidth fees and things like bandwidth alliance but they sure do a hold my beer on hypocrisy.
  - MrStonedOne 5y ago
    and what doesn't make sense about that, is in some respects argo reduces load on their network, because your origin makes a single outgoing connection to cf, that is then used to route traffic to you, reducing the effect of 3-6 packet tcp/tls connection start.
clvx 5y ago
I wonder if this could be taken as IE on Windows by default. Would this be anti competitive knowing there are industry efforts to benefit customers on other big customers.
StratusBen 5y ago
We consistently hear from our customers that egress and transit costs are one of the most difficult things to track from a cost perspective as it relates to AWS. If anyone is struggling to see egress as it relates to S3 specifically, we put together this guide here which may be relevant for this conversation as well: https://www.vantage.sh/blog/how-to-see-s3-bucket-request-metrics-egress-and-costs https://www.vantage.sh/blog/how-to-see-s3-bucket-request-met...
snicker7 5y ago
I speculate that the reason why Amazon charges so much for egress (other than to nickel-and-dime its customers) is that it inhibits hybrid-cloud or multi-cloud architectures. It promotes lock-in. In literally every business domain, Amazon succeeds not because of "customer obsession", but because of their willingness to engage in anti-competitive or outright monopolistic practices.
- slownews45 5y ago
  Interesting to see them grow so big then - they used to be the minnow to the Walmarts / Targets etc. At least in my experience (retail product ordering) their service flat out IS head and shoulders above much of the compeitition. I order off a Facebook ad - I get a drop shipment from china that takes forever, is effectively not returnable etc etc. I order from Amazon - delivery is insanely reliable, when I have an issue refunds are handled incredibly well. Even special cases (I had a weird issue with an iphone order that required a manual override and refund despite what looked like a failure to return - I got a real person and a refund promptly - they would have been relatively justified NOT to refund but I thought I'd ask). Even on AWS my support contract get's real responses. When I had a billing / customer service issue - a real person (sounded like native English speaking US call center) handled it very well. My brokers is constantly experiencing "unexpected call volume" (hour long waits). The IRS when I call has "unexpected call volume" and hour long waits. Calling southwest they have "unexpected call volume" and hour long waits. A ton of places make it virtually impossible to actually call and talk to anyone (much smaller than Amazon BTW) - you can search website all day long. Or you are speaking to call centers that can't do anything at all. I contrast AWS to the treatment we got as a paying google suite customer (endless different sales people calling us, but you don't have good paths in to trouble shoot weird state issues etc beyond basic customer service). And AWS devices support employees work calendars (via gsuite) while google devices DO NOT!
  - vel0city 5y ago
    > I contrast AWS to the treatment we got as a paying google suite customer Did you have an additional paid support contract with that Google Suite/Workspace account? What would your customer service have been with AWS if you didn't have that additional support contract? If you're paying for premium support on one service it seems a little unfair to compare that to the included base tier support of the other.
rojoroboto 5y ago
This blog post is pretty disappointing. While true that AWS's egress can be costly, it is interesting that this article makes no mention of the fact that "non-partnership egress" on GCP is actually more expensive than AWS and that Azure is roughly on par. This article misses the opportunity to talk about cloud providers in a broad context and instead attempts to publicly shame AWS for not cutting a partner discount with them on egress for customers. Don't get me wrong, I'd love an egress discount, but this is "partner channel propaganda" masked as an AWS call-out.
- deleted 5y ago
  [deleted]
- xmodem 5y ago
  This take is repeated a few times in this thread, but I'd counterpoint that AWS is the market leader, and Azure/GCP would be under much more competitive pressure to lower their bandwidth pricing if AWS brought theirs into line.
  - rojoroboto 5y ago
    This article isn’t about reducing the cost of egress though, its about giving a discount when you peer with other partners. Its a very self serving argument.
andrewguenther 5y ago
AWS Egress - $0.09/GB Azure Egress - $0.0875/GB GCP Egress - $0.11/GB Alibaba Egress - $0.123 Interesting that no mention is made of the other cloud providers pricing here when AWS isn't even the most expensive egress...Oh! I wonder if it is because Cloudflare has data transfer agreements with GCP, Azure, and Alibaba! But that would mean this is a shoddy hit piece to strong arm AWS into an agreement with them. Nonsense, I'm sure Cloudflare would never do that... Edit: I've removed claims about Cloudfront's pricing since it isn't actually clear what they charge for egress and seems to vary on a number of factors. Thanks Dylan16807 for pointing that out.
- fireant 5y ago
  The link you have submitted is about Workers however, which is a Cloudflare's function-as-a-service solution. I could not find egress pricing for general bandwidth that does not go through Workers last time I tried. Could you please point me to a document that says CF general egress pricing is $0.045 now?
  - andrewguenther 5y ago
    The original version of my post called out that you cannot actually see Cloudflare's pricing without an account, that link was the best I could find. Funny to be calling out their competitors for pricing when they won't even publish theirs...
    - Dylan16807 5y ago
      https://developers.cloudflare.com/workers/platform/pricing https://developers.cloudflare.com/workers/platform/pricing Edit: I was quoting a 100MB per request limit but that's actually ingress, egress is 512MB/unlimited so if you calculate that out it would be an even smaller fraction of a penny per terabyte for bundled workers.
    - NicoJuicy 5y ago
      Cloudflare doesn't do pricing for bandwidth, as far as I know. https://www.quora.com/How-can-Cloudflare-not-charge-for-bandwidth-for-their-CDN-capabilities https://www.quora.com/How-can-Cloudflare-not-charge-for-band...
erhk 5y ago
No one will migrate off baremetal if ingress were to be that expensive.
torcete 5y ago
Base on the costs of transferring data into AWS being much cheaper, I wonder if a reverse transfer protocol could be designed with an adaptation of binary search. You divide your data in blocks of, let's say 100 Mb, the receiver creates a block of random data and sends it to a process running on AWS. The process has to answer, higher, lower or exact (and then both parties move to the next block). Probably it will be very slow, but it will lower a lot the bandwidth usage for egression.
- CodesInChaos 5y ago
  Getting out more data than bandwidth allows is a information theoretical impossibility. Your proposal has the flaw that it needs one pass per bit in the block on average.
mindfulplay 5y ago
Great post. Minor rant: However while Cloudflare does not charge for edge-egress, their pricing story is terrible, awful and dare I say terrible again. It looks like they took their internal evolving developer docs and turned it into pricing for customers. There is a tree of docs and pricing you have to get to the punchline which is that Cloudflare for the most part is several orders of magnitude cheaper than AWS or any other cloud provider. And each 'team' seems to have their own little pricing page with arcane details - exactly the definition of 'ship your org chart as products'. Just check out Workers pricing for instance. The product is great but the pricing story is silly. But it's almost like they want to drive customers directly to AWS with Amazon's simple pricing calculator and tools.
- agucova 5y ago
  In what sense is it order of magnitudes cheaper? Egress traffic?
  - mindfulplay 5y ago
    If you are serving static assets then essentially it's pretty cheap. You can use KV / Workers and then use the caching to minimize cost which measures requests not bandwidth.
pqdbr 5y ago
If you are in Brazil, AWS Egress is so expensive (and the BRL is so undervalued against the USD right now) that hosting in a bare metal with generous free bandwidth tiers (like 20TB you can find pretty much anywhere) is a huge competitive advantage against those that are hosting on AWS.
IceHegel 5y ago
Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers and the same math shows cloudflare with a ~3800% markup. From u/andrewguenther: AWS Egress - $0.09/GB Azure Egress - $0.0875/GB GCP Egress - $0.11/GB Alibaba Egress - $0.123 Cloudflare - $0.09/GB pre-April 2021 $0.045 now 1. If AWS is charging me $6 per Mbps for something that costs them 8¢, why is Cloudflare charging me $3 for the same 8¢ good? 2. What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down?
- baskire 5y ago
  Buying 1 Mibps means you can use up to 1Mibps. Paying $/GiB means you can use 100s of Gibps for a split second and only pay the $/GiB fee. In the past i did a TCO comparison for a large spikey workload where the $/GiB cost even at list was cheaper than allocating enough bandwidth for peak events. At small scale the $/Mibps likely comes out cheaper, but for any large company sitting on 100s of Gibps of excess capacity for an event that occurs once or twice a year, or a few hours per day is very costly and might eat up savings from p99/p95th billing.
- Dylan16807 5y ago
  > Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers None of the biggest ones are trying to compete on egress. Especially because, why make it easy to run some of your services outside their cloud? Please move it all in to the same place and dedicate your budget to them. > Cloudflare - $0.09/GB pre-April 2021 $0.045 now That's not general egress, that's one random service optimized for using more CPU time. If you use the less-CPU-time version, bandwidth is free/negligible. > What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down? Better tech allows vastly increasing amounts of data to flow through the same wires using the same amount of equipment. The price per unit goes up, but the price per byte goes down.
trinovantes 5y ago
Weird that Digital Ocean is not part of the bandwidth alliance. If they were, would that mean I may not get charged bandwidth for traffic from my DO VPS to CF proxies?
- ksec 5y ago
  Oh that is interesting because I remember DO and Linode were part of their partners during the initial announcement. And now its name is gone. https://www.cloudflare.com/en-gb/bandwidth-alliance/ https://www.cloudflare.com/en-gb/bandwidth-alliance/
eloff 5y ago
Thank you cloudflare for calling AWS out on their ridiculous transfer pricing. Anybody working in this space has been frustrated with it for over a decade. We all know it's a ripoff, and we all know it's priced in a way to lock you and others into AWS. I've used AWS at every job I've ever had. But that bandwidth pricing has been incredibly frustrating. Also where possible we now host on lightsail, because the bandwidth savings make the hardware completely free. It's not the best hardware, but it works. The web console is also a joy to use by comparison.
- Terretta 5y ago
  Or, it's priced at market -- within 10% of Azure, GCP, and Alibaba, and not the highest priced among them.
  - eloff 5y ago
    Except it's not when it comes to peered bandwidth, like with cloudflare. Which is a big deal especially to cloudflare. If you're in Alibaba, Azure or Google Cloud you can cut your costs by putting cloudflare in front, even though that adds another service you pay for. And all of them have ridiculous bandwidth pricing compared to the entire rest of the hosting industry.
    - Terretta 5y ago
      Calling the pricing ridiculous is why engineers are underpaid. Don’t value bottoms up, you undersell yourself. Value top down.
inopinatus 5y ago
TL;DR: Cloudflare are upset that AWS won’t peer with them in terms of Cloudflare’s choosing, and this is the angry shitpost.
- agucova 5y ago
  It does seem like Cloudflare are not the only ones upset about how AWS handles traffic
soheil 5y ago
This is a bit shameful. Publicly ousting AWS because they basically don't want to join Cloudflare's little attempt at market domination named, in 1984 style, Bandwidth Alliance. Any attempt AWS makes or for that matter any company makes to build a moat around a business can be categorized as anti-consumer behavior. Of course, AWS should do their best to make sure they make it harder for their customer to leave on a dime while at the same time doing their best to keep them happy. Why wouldn't any business do that? It's not a charity. The entire cloud infrastructure providers could be seen the same way, you build your infra in the cloud and it'll become significantly more difficult to move away to your own datacenter at some point in the future. This is well know yet, that's not an issue raised here but asymmetric bandwidth cost is the axe Cloudflare chooses to grind? It seems to be an egregious and opportunistic move by Cloudflare to go after AWS right as their parent CEO steps down after a divorce.
mikevm 5y ago
Speaking of egress... I've been thinking about hosting stuff on Hetzner and using Backblaze B2 for storage to get a cheap alternative to EC2+S3, but unfortunately they're not partners so I won't get free traffic, or low latency :(.
- namibj 5y ago
  They (B2) are with Packet (rebranded to Equinix Metal), last I checked.
- Wallacy 5y ago
  If tou put the CF in between will not pay the egress. But will be better to Hetzer be part of the alliance.
AnalyticBoffin 5y ago
I would point out that Oracle's Egress is $0.008 per GB, based on their estimator. That makes a hybrid cloud (on-premises/cloud) more than doable -lots more flexibility. The rest of the cost structure is transparent, with industry leading pricing and performance (take a look at HeatWave for an example of this). As a cloud advisor, I often see a lack of proper due diligence in cloud adoption decisions. The typical cloud decision is made at the CFO or CIO level without full visibility into the cost of complex application delivery. These are not stupid people. My take is that a deliberate fog is created with many cloud vendors to obfuscate the true cost of business. Three blindsides hit the executives. One, performance at production scale require ala carte additions for reasons outside of the workload's demands. A case in point, a cloud vendor required an 8 vCPU instance to equal the performance of a four core, on-premises box at production workloads. The storage driver ate half of the available CPU and IO traffic was routed over a single vNIC shared with application traffic. This was unexpected by the customer -something hidden in the vendor's implementation. Two, lack of transparency, leads to billing surprises. Another of my clients budgeted $45k for a 3 month production cloud deployment test. The CFO had an anti-pleasant meeting with the CIO when the first month's bill hit $40k. The cause was egress charges. The experiment ended and they stayed on-premises. At least they had tried the workload at production scale. The third is cloud 'divorces' are almost never considered. The cost of 'divorce' is extremely high, cost prohibitive in many cases but not just for egress fees. For instances a client using a cloud native database found initial performance good for inserts, but after time query performance became unacceptable. The promised scaling was not there. Fortunately the lesson was learned while egress was 'affordable' for moving 2TB of production data out. The pain came into play with the additional computing resources needed to export (not copy out) the data out for repatriation. Insult was added by maintaining two production environments for the 10 days to perform the export and required expertise to implement it. That was for one application's analytic database. Now imagine moving full technology stacks for multiple critical application. How would I avoid the issues: 1. If billing isn't fully transparent, then move to another vendor. Too often, a small application gets to a $100k annual spend, the cloud vendor give a fair discount at this point. Once the spend hits a $1m, usually to compensation for vendor deficiencies at production scale, there is no additional assistance offered. The vendor has you exactly where they want you and your are locked. Greater discounts for 3 year terms you say, you've got three years of sub par service and get to spend more money to solution yourself out of the hole. 2. Test at production workloads. Does the spend match what you need or do you need ala carte upgrades to get you where your workload's performance needs to be today and out years? On then other hand are you paying for the architectural deficiency of the vendor compared to your current as built. 3. If you are dissatisfied with the quality of product or more importantly the level of service, can you just leave then and there? A quality cloud vendor will create a great experience pre and post sale. The will not create artificial barriers to departure. The golden rule for a valued cloud vendor is, "I have to prove myself, every day, in every way to keep my customer for tomorrow." If you sense that isn't the demonstrated action of a perspective or current cloud vendor -move along.
Rd6n6 5y ago
Digital oceans bandwidth charges are about 0.01/GB, which is the cheapest I’ve found. Anybody know of similar or better offerings elsewhere?
- mritzmann 5y ago
  Hetzner Cloud is known in the DACH region (Germany, Austria, Switzerland). They charge 1.19€ ($1) per TB and 20TB per Compute/VM is included.
- neop1x 5y ago
  Scaleway - egress free
reilly3000 5y ago
In AWS's defense, their egress pricing covers basically 100% of the cost of networking for their whole platform. VPC, ingress, and many other features and services roll up under that bill. On that basis I have to disagree with the analysis in this post. However, their main point is well-made, its a lopsided agreement that traps customer's data where in some cases its just not affordable to leave. Its not well known to customers until they face the bill... there was a story of a giant data project at NASA where they were planning to store hundreds of petabytes of data on AWS, but neglected to account for the costs of egress which through the entire project into peril: https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/ https://www.theregister.com/2020/03/19/nasa_cloud_data_migra...
mark_l_watson 5y ago
I only use AWS because companies I work for are locked in. When I am spending my own money, I love the low costs at Hetzner and when I need to spin up VPSs I find GCP much more convenient to use. I am also evaluating Oracle right now because they have a generous free tier VPS and their services are easy enough to use. I pay $$$ for Apple’s products and services, but I don’t enjoy using AWS. I do enjoy Amazon’s sales and delivery platform for buying physical things, and their digital media, so I don’t mind paying $$ for that.
xtat 5y ago
While it ain't wrong, I'm not sure I want to trade one cartel for another, Cloudflare.
mdavis6890 5y ago
This post is not very fair, and seems to have started with a conclusion "big company screwing their customers," instead of really digging in to learn why things are the way they are for good non-customer-screwing reasons. The post DOES actually call out that wholesale bandwidth is symmetric, but then jumps to the OPPOSITE of the correct interpretation of that: That AWS's pricing to the end customer should be equally symmetric, and the fact that it isn't is evidence of AWS's customer-screwing behavior. The actual reason is that AWS's customers, like those of other cloud providers, are running externally-facing services that mostly send data OUT to their users. Given the inherent symmetry in wholesale bandwidth, it means that AWS has to build out based on their total egress bandwidth. As customers egress more data their users (or wherever/whoever), AWS has to not only pay for more wholesale bandwidth, they also have to build out all the additional infrastructure (routers, etc) to support it. It's very expensive. The ingress bandwidth, being less than the egress bandwidth already noted above, therefore is effectively free to AWS, and they can charge very low rates to their customers for it.

742 points by jgrahamc 5y ago

Does this make sense? He seems to be comparing peak bandwidth pricing (megabits per second) to average throughput pricing (megabits). Peak bandwidth pricing is really a lie (if you actually used it all the time you'd be looking for a new host)

eastdakota 5y ago
Calculated assuming 20% utilization, which is the low end of industry average (20% – 40%).

cameronh90 5y ago

"In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher. We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn't seem to be an innocent explanation. " One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that?

sitkack 5y ago
> One consideration could be that you can't directly control data ingress. Are you thinking UDP? Or for a DDOS? Connection setup overhead could be accounted for. I highly doubt this is the reason.
- linuxftw 5y ago
  If there's an IP exposed on the internet, you can just send it tcp payloads. The end destination will silently drop them, but it doesn't mean people can't send you gigs of useless data.
  - otterley 5y ago
    I think you mean UDP. TCP requires a 3-way handshake first.
    - ithkuil 5y ago
      Intermediate routers don't care about that; they only forward the IP packets; four target host/firewall will drop them (because they don't belong to a valid connection) but they will be still accounted for as ingress traffic.
      - otterley 5y ago
        Correct, but you can't get much bandwidth through until the 3-way handshake is completed. Sending a bunch of unanswered SYN packets isn't really a great way to instigate a DDoS, compared to sending avalanches of 64KiB UDP packets.
        ithkuil 5y ago
        As long as there is no connection tracking you can send whatever crap you want, including replayed packets from the middle of a connection, perhaps even huge packets with a syn flag ... As long as the accounting happens before a firewall performs basic TCP sanity checks you're going to pay for it
        kazen44 5y ago
        in some setups, you could send non IP packets (especially in MPLS fabric if accounting happens by LSP).
- blamethenetwork 5y ago
  At a routing and peering level. Once you have an announcement for your netblock out there, traffic will start to head towards it. A lot of this is due to the BGP Path Selection Algorithm. You can try and influence how traffic arrives, by doing things like, AS prepends, but you are still going to get traffic. The main reason for this is that the other side that is egressing to you has their own egress policy that also follows path selection. Things like localpref and weight will force my traffic to leave via a path before it considers how a network has AS padded. As an example: Lets say I want to egress (company A) to a downstream company (company B). If I learn routes to Company B via multiple ways: peering fabric (low cost), paid peering (medium cost), transit1 (high cost, variable quality), transit2 (low cost variable quality), I can choose which way my traffic goes, via localpref, weight etc. Only when I view the paths equally (equal localpref, weight etc.) will I evaluate the shortest AS Path (which the receiving company has influence on). The only way to completely not get inbound traffic via a specific link, is to remove your BGP advertisement for your netblock from that link. (some providers also let you do this selectively via BGP communities). There are also some other tips/tricks - such as adding a more specific prefix to a certain link, to attract traffic, but care needs to be made to have a fallback route in case things go wonky.
tw04 5y ago
Coming from someone who believes that the AWS egress fees are highway robbery, I'd throw this out there. In order to maintain settlement free peering, you need to keep egress and ingress somewhat even with your peers. Amazon proper probably has 3x egress vs ingress between prime and twitch. AWS is an easy avenue to balance it out. All that being said: I would guess there's almost 0 chance that's what's happening here. Amazon is just using it as a form of lock-in IMO.
- hnpasf 5y ago
  I agree that it is just lock-in. The price of egress is the same even when using devices like the Snowball where the data is loaded onto a server and physically shipped to you.
qeternity 5y ago
> One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. Most request/response models are very asymmetric. I can request a relatively large asset from S3 with a small request and maliciously generate fees as an attacker. AWS egress is vendor lock-in 100%.
eli 5y ago
It's also just a way to segment customers. The people doing tons of egress, on average, probably have bigger budgets for cloud computing. There's no particular reason AWS costs have to mirror how they charge.
res0nat0r 5y ago
The only service AWS hasn't consistently lowered pricing on (unlike almost everything else) is their egress cost. Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board. But egress is the exception. They're keeping pricing high for this exact reason to prevent folks from moving their data elsewhere. I've got about 25TB of stuff I'd love to keep all in Glacier Deep Archive, but the restore costs are just too insane to justify due to the egress pricing still at .09c after all these years. Too bad.
- skuhn 5y ago
  I view network egress as the primary lock-in mechanism for all of AWS. If you want to migrate to another provider, network egress costs mean that you'll spend multiples of your normal monthly operating costs to do so. That stifles competition. But well before that point, most services in AWS have already been built around avoiding network egress wherever possible. You're always going to prefer AWS APIs and services, even if they aren't the best for your use case, because services outside AWS have a network egress tariff placed on them. So you aren't always buying best-of-breed, you're buying the best of what AWS chooses to offer (or the selection of vendors that choose to build in AWS to remain competitive). And if that isn't good enough, your only other option is to migrate out (and pay those egress costs!).
- qeternity 5y ago
  > Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board. How would keeping prices high for these services keep people locked in? The whole point of vendor lock in is there is some other indirect cost that you bear by switching, which is why you don't switch. For AWS, this indirect cost is bandwidth. It is the lock in mechanism.
  - res0nat0r 5y ago
    > How would keeping prices high for these services keep people locked in? By doing the opposite and raising prices across the board randomly because they know folks can't spend the time and money to migrate without a large support burden or time/money sunk cost. The only service not going down in price is egress and the only thing I've seen which has been like this, to directly discourage folks from migrating their data to say GCP when their offering(s) look more attractive.
isbvhodnvemrwvn 5y ago
If you get DDoSed you also pay through the roof in fees (for whatever is in your VPC - e.g. ELB which are billed per throughput) unless you buy their anti-DDoS services (AWS Shield Advanced) for 3k$/mo - without that AWS won't waive the fees.
RKearney 5y ago
> One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that? An equally likely reason is by making ingress free it helps balance out ingress and egress traffic flows which are usually desirable for settlement-free peering agreements.
kevincox 5y ago
The simple explanation is that ingress is far, far lower than egress so the cost isn't relevant since links tend to be symmetrical. So the cost to them is determined based on egress, their ingress provisioning happens at no additional cost.

thejosh 5y ago

Azure discount for Cloudflare seems tiny though, but that won’t fit into their recent marketing efforts Cloudflare has been doing.

pumanoir 5y ago
I didn't see the amount of the discount in the article. Is there a place where one can see the pricing (i.e. 10TB azure -> CF)?
- donmcronald 5y ago
  I couldn’t find it when I searched. Some BA participants have fair usage limits too. Ex: I think Wasabi is about 1x your stored data per month.

kbuck 5y ago

I've assumed this was well-known... AWS has never been a good deal on the billing side. You're paying a huge premium for API access and the strength of the AWS brand. I was a bit confused as to why Cloudflare was blogging about it until halfway through the article, when The Bandwidth Alliance came in. Makes sense now: they just want to shame Amazon into offering free egress bandwidth to Cloudflare. This is basically a hit piece. Also, another aside: for hosting providers (AWS included), ingress bandwidth is typically free because it's also basically free for the provider. Outbound dominates so much that the inbound traffic is a rounding error if it's billed by their transit providers at all. But I agree that the high egress prices are a vendor lock-in strategy.

deleted 5y ago
[deleted]
pkilgore 5y ago
Perhaps I misunderstood, but the article says Amazon already does not pay for egress to Cloudflare. But Amazon still charges their customers as if they did. Did I misunderstand something?
- inopinatus 5y ago
  You sure did. Peering isn’t free, or even cheap. Cloudflare are whining for special treatment for their bandwidth cartel, and AWS are large enough to tell them to get fucked. The key is realising that Cloudflare have always been about disingenuously leveraging other people’s stuff, since they’re basically just a middleman.
  - Dylan16807 5y ago
    Peering's definitely cheap compared to what Amazon's charging. Amazon even has a peering product for 16.5 cents per Mbps, but it's not usable in this circumstance. "Please join our free peering group of many companies." is neither a cartel nor a request for special treatment. What are they "disingenuously leveraging"?

edoceo 5y ago

Reads like CF has a bone to pick with AWS. Are they mad about AWS not in the peering group? Is the AWS price really out of line - the comparison to wholesale cost only seems to miss a lot of infrastructure that gets baked into the final retail prices - it's not simply (Retail-Wholesale=7000% margin). Feels like a weak "hit piece" to me.

notsureaboutpg 5y ago
Cloudflare is ramping up it's enterprise sales and deals (and doing a good job of it too, anecdotally). So they are competing more and more with AWS (their Workers product is a good example), and why not point out things people dislike about their competitors?
Daishiman 5y ago
I think it's quite fair. You're not paying anywhere near close to the price of the infra with AWS; you're paying for the convenience, but be mindful that the cost has a break-even point and many companies may be beyond that point. I don't have the experience to assess whether the total cost of ownership is favorable to most AWS clients or not.
pkilgore 5y ago
They directly address these costs in the article as "rounding error". Do you have some evidence to dispute that? Did I misunderstand them?
isbvhodnvemrwvn 5y ago
They are in the market for reducing that bandwidth by caching stuff at CloudFlare, so it's not exactly surprising.

AgentK20 5y ago

Working in realtime games that require high bandwidth usage, AWS and basically every other public cloud is fundamentally unusable for us because of this exact problem. Our bare metal infrastructure for the Hypixel Minecraft network uses 3-4PB per month, so we simply lease a 100gbps transit link billed at 95th percentile. Last I checked, AWS wanted ~$200k per MONTH, for just bandwidth (no compute, memory, storage, or anything else). We'd love to be able to use the cloud, but not at the cost of increasing our monthly expenses by an entire order of magnitude, so we just stick to bare metal colocation. I honestly believe that if you have the technical skill in-house, and are spending more than, say, $30k/mo on public cloud hosting, that you should seriously evaluate whether bare metal could significantly decrease your costs.

aclelland 5y ago
At $JOB, we have a similar issue with static content. We serve over 1PB a month and the price AWS would charge to use CloudFront is an order of magnitude larger than even a Cloudflare Enterprise plan (which comes with some nice bells and whistles that Cloudfront doesn't offer). Even with discounts from AWS it just doesn't make sense to use AWS to serve up the assets to users. We do use S3 as our static asset backend and the combination works really well. I would love to see Cloudflare release a S3 compatible storage service though. I think we'd jump onto that in a heartbeat.
- HeavenFox 5y ago
  Backblaze has S3-compatible API and they have free data transfer to Cloudflare
  - aclelland 5y ago
    The last time I looked at BB S3 API they didn't offer the lifecycle controls that AWS over. Mainly the ability to remove old versions of files after a period of time and we didn't want to roll our own expiry solution. Might be worth looking again though.
    - 5y ago

api 5y ago

It's very much by design. They want ingress not egress to create "gravity." The end game is for virtually all compute to happen in the cloud and be accessed only by thin client. This pricing drives things in this direction by making companies that would use the cloud to just service local compute less profitable than companies that host everything in the cloud and provide only web or other thin client access. There are providers that don't do this, but they are much less batteries-included. The best bandwidth deals are via bare metal hosting which is basically just box-in-a-rack rental. You can get bandwidth there by capacity rather than transfer quota. It's many many orders of magnitude cheaper.

derefr 5y ago

> The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first. Presuming the average AWS customer is running an API SaaS, said customer’s API response load (= egress bandwidth) will tend to be a good measure of how much “useful work” said SaaS is doing for its own clients — either in the form of API responses, requests to third parties, emails sent, notifications pushed, uploads to other providers, etc. As such, egress bandwidth tends to be a good measure of having product-market fit, and thereby of willingness-to-pay. Things that trigger mostly ingress load — dropping attacks at a firewall, accepting uploads, receiving emails/notifications, etc. aren’t as emblematic of product-market fit (they aren’t things you would traditionally charge customers for “by volume”), but rather are usually activities done as loss-leaders; and so do not correlate to willingness-to-pay.

alvis 5y ago

Back in the old day, we were charged by the maximum bandwidth. Not quite scalable maybe, but it definitely won’t bankrupt you if you’re not careful.

mdasen 5y ago

On the Hotel California Effect If it's expensive to move data out of AWS, it's not just about making it hard for customers to leave AWS. It means that any third-party service that wants to sell to AWS customers must also use AWS. For example, Snowflake can't really run its own data centers. They need to rent from AWS. If you're a company on AWS, you won't want to spend $0.08/GB sending things to Snowflake. The high egress cost gives Amazon a lot of power to keep the third-party ecosystem within AWS and without as much negotiating power on things like EC2 costs. Snowflake can't say to Amazon, "if you don't give us a great discount, we're going to move our operations to DigitalOcean" because Amazon would just say, "no you're not. Your customers aren't going to pay $80/TB to load things into Snowflake when you're charging them $23-40/TB for storage. They'd be paying 2-3.5x your storage costs just to load the data into your system!" Yes, I'm sure that Snowflake does get a discount from AWS, but having less negotiating power can make discounts smaller. There are other reasons to want a third party to use AWS if you're on AWS. Still, the egress pricing seems to make it very hard for third-party tech providers not to use AWS if their customers are using it.

Hokusai 5y ago
Amazon is really good at leveraging its semi monopolistic presence in different markets. Once you are big enough you only need to tweak a few parameters to dominate the market. It's interesting how AWS uses egress to this effect. Your comment is very insightful.
- stadium 5y ago
  They employ teams of economists to model price elasticity of demand. There are likely many orders of magnitude more than a few parameters. And they look at how a change in one product feature or pricing model cannibalizes or compliments other features and prices.
nindalf 5y ago
Thank you for this. I’m surprised that CloudFlare didn’t mention it in an otherwise thorough post. After all, the reason they’re trying to pressure AWS to change is that they’d like to offer services to AWS customers without bandwidth markup. Essentially competing with existing AWS services on an even footing. This to me makes it unlikely that AWS will change their bandwidth pricing. If it was just about this revenue source disappearing, they might bite the bullet and hope that the lower prices would attract more customers in the long term. But bandwidth pricing is the moat around their castle. If it didn’t exist, Snowflake and others would leave. But more than that, new AWS services benefit from the captive audience choosing them by default. Without the moat, each service has to compete on its merits. AWS Lambda, for example would need to be better than CloudFlare Workers or fly.io containers. It won’t win customers like it does today just because it’s “free” bandwidth. Removing these fees is a risk AWS is currently choosing not to take. And they won’t change their pricing regardless of the number of blog posts that competitors write. The only thing that would force a change is an exodus of customers leaving AWS citing this as a reason.

donmcronald 5y ago

It looks like Cloudflare cut egress pricing for workers unbound to $.0045 per GB, but that’s still half of AWS or “40x markup” for me in Canada. The original workers model of pay per execution is incredible. IIRC I figured one run is the equivalent of 6KB of egress at AWS. Is there a point where Cloudflare will start charging egress on those if I transfer too much? I want to use it for something, but don’t like the “free bandwidth until you hit the invisible limit” side of things Cloudflare has going. 1. https://blog.cloudflare.com/workers-unbound-ga/ https://blog.cloudflare.com/workers-unbound-ga/

satyrnein 5y ago
There's a typo, it's $0.045 per GB (half of AWS, as mentioned). I got excited for a second!
- donmcronald 5y ago
  Damn. I'm stupid today. I don't think I can edit anymore either :-(
deleted 5y ago
[deleted]
yepcfstyle 5y ago
Cloudflare argo pricing (basically stay on the CF network) is almost identical to GCP Premium Tier and AWS default tier. They claim it does some cool optimizations to traffic path, no different to BGP optimizers - it is simply ridiculous to presume GCP/AWS doesnt optimize their routes too on a continous basis - yet apples and oranges doesn't apply CF says. I applaud CF nudging AWS to reduce bandwidth fees and things like bandwidth alliance but they sure do a hold my beer on hypocrisy.
- MrStonedOne 5y ago
  and what doesn't make sense about that, is in some respects argo reduces load on their network, because your origin makes a single outgoing connection to cf, that is then used to route traffic to you, reducing the effect of 3-6 packet tcp/tls connection start.

clvx 5y ago

I wonder if this could be taken as IE on Windows by default. Would this be anti competitive knowing there are industry efforts to benefit customers on other big customers.

StratusBen 5y ago

We consistently hear from our customers that egress and transit costs are one of the most difficult things to track from a cost perspective as it relates to AWS. If anyone is struggling to see egress as it relates to S3 specifically, we put together this guide here which may be relevant for this conversation as well: https://www.vantage.sh/blog/how-to-see-s3-bucket-request-metrics-egress-and-costs https://www.vantage.sh/blog/how-to-see-s3-bucket-request-met...

snicker7 5y ago

I speculate that the reason why Amazon charges so much for egress (other than to nickel-and-dime its customers) is that it inhibits hybrid-cloud or multi-cloud architectures. It promotes lock-in. In literally every business domain, Amazon succeeds not because of "customer obsession", but because of their willingness to engage in anti-competitive or outright monopolistic practices.

slownews45 5y ago
Interesting to see them grow so big then - they used to be the minnow to the Walmarts / Targets etc. At least in my experience (retail product ordering) their service flat out IS head and shoulders above much of the compeitition. I order off a Facebook ad - I get a drop shipment from china that takes forever, is effectively not returnable etc etc. I order from Amazon - delivery is insanely reliable, when I have an issue refunds are handled incredibly well. Even special cases (I had a weird issue with an iphone order that required a manual override and refund despite what looked like a failure to return - I got a real person and a refund promptly - they would have been relatively justified NOT to refund but I thought I'd ask). Even on AWS my support contract get's real responses. When I had a billing / customer service issue - a real person (sounded like native English speaking US call center) handled it very well. My brokers is constantly experiencing "unexpected call volume" (hour long waits). The IRS when I call has "unexpected call volume" and hour long waits. Calling southwest they have "unexpected call volume" and hour long waits. A ton of places make it virtually impossible to actually call and talk to anyone (much smaller than Amazon BTW) - you can search website all day long. Or you are speaking to call centers that can't do anything at all. I contrast AWS to the treatment we got as a paying google suite customer (endless different sales people calling us, but you don't have good paths in to trouble shoot weird state issues etc beyond basic customer service). And AWS devices support employees work calendars (via gsuite) while google devices DO NOT!
- vel0city 5y ago
  > I contrast AWS to the treatment we got as a paying google suite customer Did you have an additional paid support contract with that Google Suite/Workspace account? What would your customer service have been with AWS if you didn't have that additional support contract? If you're paying for premium support on one service it seems a little unfair to compare that to the included base tier support of the other.

rojoroboto 5y ago

This blog post is pretty disappointing. While true that AWS's egress can be costly, it is interesting that this article makes no mention of the fact that "non-partnership egress" on GCP is actually more expensive than AWS and that Azure is roughly on par. This article misses the opportunity to talk about cloud providers in a broad context and instead attempts to publicly shame AWS for not cutting a partner discount with them on egress for customers. Don't get me wrong, I'd love an egress discount, but this is "partner channel propaganda" masked as an AWS call-out.

deleted 5y ago
[deleted]
xmodem 5y ago
This take is repeated a few times in this thread, but I'd counterpoint that AWS is the market leader, and Azure/GCP would be under much more competitive pressure to lower their bandwidth pricing if AWS brought theirs into line.
- rojoroboto 5y ago
  This article isn’t about reducing the cost of egress though, its about giving a discount when you peer with other partners. Its a very self serving argument.

andrewguenther 5y ago

AWS Egress - $0.09/GB Azure Egress - $0.0875/GB GCP Egress - $0.11/GB Alibaba Egress - $0.123 Interesting that no mention is made of the other cloud providers pricing here when AWS isn't even the most expensive egress...Oh! I wonder if it is because Cloudflare has data transfer agreements with GCP, Azure, and Alibaba! But that would mean this is a shoddy hit piece to strong arm AWS into an agreement with them. Nonsense, I'm sure Cloudflare would never do that... Edit: I've removed claims about Cloudfront's pricing since it isn't actually clear what they charge for egress and seems to vary on a number of factors. Thanks Dylan16807 for pointing that out.

fireant 5y ago
The link you have submitted is about Workers however, which is a Cloudflare's function-as-a-service solution. I could not find egress pricing for general bandwidth that does not go through Workers last time I tried. Could you please point me to a document that says CF general egress pricing is $0.045 now?
- andrewguenther 5y ago
  The original version of my post called out that you cannot actually see Cloudflare's pricing without an account, that link was the best I could find. Funny to be calling out their competitors for pricing when they won't even publish theirs...
  - Dylan16807 5y ago
    https://developers.cloudflare.com/workers/platform/pricing https://developers.cloudflare.com/workers/platform/pricing Edit: I was quoting a 100MB per request limit but that's actually ingress, egress is 512MB/unlimited so if you calculate that out it would be an even smaller fraction of a penny per terabyte for bundled workers.
  - NicoJuicy 5y ago
    Cloudflare doesn't do pricing for bandwidth, as far as I know. https://www.quora.com/How-can-Cloudflare-not-charge-for-bandwidth-for-their-CDN-capabilities https://www.quora.com/How-can-Cloudflare-not-charge-for-band...

erhk 5y ago

No one will migrate off baremetal if ingress were to be that expensive.

torcete 5y ago

Base on the costs of transferring data into AWS being much cheaper, I wonder if a reverse transfer protocol could be designed with an adaptation of binary search. You divide your data in blocks of, let's say 100 Mb, the receiver creates a block of random data and sends it to a process running on AWS. The process has to answer, higher, lower or exact (and then both parties move to the next block). Probably it will be very slow, but it will lower a lot the bandwidth usage for egression.

CodesInChaos 5y ago
Getting out more data than bandwidth allows is a information theoretical impossibility. Your proposal has the flaw that it needs one pass per bit in the block on average.

mindfulplay 5y ago

Great post. Minor rant: However while Cloudflare does not charge for edge-egress, their pricing story is terrible, awful and dare I say terrible again. It looks like they took their internal evolving developer docs and turned it into pricing for customers. There is a tree of docs and pricing you have to get to the punchline which is that Cloudflare for the most part is several orders of magnitude cheaper than AWS or any other cloud provider. And each 'team' seems to have their own little pricing page with arcane details - exactly the definition of 'ship your org chart as products'. Just check out Workers pricing for instance. The product is great but the pricing story is silly. But it's almost like they want to drive customers directly to AWS with Amazon's simple pricing calculator and tools.

agucova 5y ago
In what sense is it order of magnitudes cheaper? Egress traffic?
- mindfulplay 5y ago
  If you are serving static assets then essentially it's pretty cheap. You can use KV / Workers and then use the caching to minimize cost which measures requests not bandwidth.

pqdbr 5y ago

If you are in Brazil, AWS Egress is so expensive (and the BRL is so undervalued against the USD right now) that hosting in a bare metal with generous free bandwidth tiers (like 20TB you can find pretty much anywhere) is a huge competitive advantage against those that are hosting on AWS.

IceHegel 5y ago

Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers and the same math shows cloudflare with a ~3800% markup. From u/andrewguenther: AWS Egress - $0.09/GB Azure Egress - $0.0875/GB GCP Egress - $0.11/GB Alibaba Egress - $0.123 Cloudflare - $0.09/GB pre-April 2021 $0.045 now 1. If AWS is charging me $6 per Mbps for something that costs them 8¢, why is Cloudflare charging me $3 for the same 8¢ good? 2. What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down?

baskire 5y ago
Buying 1 Mibps means you can use up to 1Mibps. Paying $/GiB means you can use 100s of Gibps for a split second and only pay the $/GiB fee. In the past i did a TCO comparison for a large spikey workload where the $/GiB cost even at list was cheaper than allocating enough bandwidth for peak events. At small scale the $/Mibps likely comes out cheaper, but for any large company sitting on 100s of Gibps of excess capacity for an event that occurs once or twice a year, or a few hours per day is very costly and might eat up savings from p99/p95th billing.
Dylan16807 5y ago
> Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers None of the biggest ones are trying to compete on egress. Especially because, why make it easy to run some of your services outside their cloud? Please move it all in to the same place and dedicate your budget to them. > Cloudflare - $0.09/GB pre-April 2021 $0.045 now That's not general egress, that's one random service optimized for using more CPU time. If you use the less-CPU-time version, bandwidth is free/negligible. > What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down? Better tech allows vastly increasing amounts of data to flow through the same wires using the same amount of equipment. The price per unit goes up, but the price per byte goes down.

trinovantes 5y ago

Weird that Digital Ocean is not part of the bandwidth alliance. If they were, would that mean I may not get charged bandwidth for traffic from my DO VPS to CF proxies?

ksec 5y ago
Oh that is interesting because I remember DO and Linode were part of their partners during the initial announcement. And now its name is gone. https://www.cloudflare.com/en-gb/bandwidth-alliance/ https://www.cloudflare.com/en-gb/bandwidth-alliance/

eloff 5y ago

Thank you cloudflare for calling AWS out on their ridiculous transfer pricing. Anybody working in this space has been frustrated with it for over a decade. We all know it's a ripoff, and we all know it's priced in a way to lock you and others into AWS. I've used AWS at every job I've ever had. But that bandwidth pricing has been incredibly frustrating. Also where possible we now host on lightsail, because the bandwidth savings make the hardware completely free. It's not the best hardware, but it works. The web console is also a joy to use by comparison.

Terretta 5y ago
Or, it's priced at market -- within 10% of Azure, GCP, and Alibaba, and not the highest priced among them.
- eloff 5y ago
  Except it's not when it comes to peered bandwidth, like with cloudflare. Which is a big deal especially to cloudflare. If you're in Alibaba, Azure or Google Cloud you can cut your costs by putting cloudflare in front, even though that adds another service you pay for. And all of them have ridiculous bandwidth pricing compared to the entire rest of the hosting industry.
  - Terretta 5y ago
    Calling the pricing ridiculous is why engineers are underpaid. Don’t value bottoms up, you undersell yourself. Value top down.

inopinatus 5y ago

TL;DR: Cloudflare are upset that AWS won’t peer with them in terms of Cloudflare’s choosing, and this is the angry shitpost.

agucova 5y ago
It does seem like Cloudflare are not the only ones upset about how AWS handles traffic

soheil 5y ago

This is a bit shameful. Publicly ousting AWS because they basically don't want to join Cloudflare's little attempt at market domination named, in 1984 style, Bandwidth Alliance. Any attempt AWS makes or for that matter any company makes to build a moat around a business can be categorized as anti-consumer behavior. Of course, AWS should do their best to make sure they make it harder for their customer to leave on a dime while at the same time doing their best to keep them happy. Why wouldn't any business do that? It's not a charity. The entire cloud infrastructure providers could be seen the same way, you build your infra in the cloud and it'll become significantly more difficult to move away to your own datacenter at some point in the future. This is well know yet, that's not an issue raised here but asymmetric bandwidth cost is the axe Cloudflare chooses to grind? It seems to be an egregious and opportunistic move by Cloudflare to go after AWS right as their parent CEO steps down after a divorce.

mikevm 5y ago

Speaking of egress... I've been thinking about hosting stuff on Hetzner and using Backblaze B2 for storage to get a cheap alternative to EC2+S3, but unfortunately they're not partners so I won't get free traffic, or low latency :(.

namibj 5y ago
They (B2) are with Packet (rebranded to Equinix Metal), last I checked.
Wallacy 5y ago
If tou put the CF in between will not pay the egress. But will be better to Hetzer be part of the alliance.

AnalyticBoffin 5y ago

I would point out that Oracle's Egress is $0.008 per GB, based on their estimator. That makes a hybrid cloud (on-premises/cloud) more than doable -lots more flexibility. The rest of the cost structure is transparent, with industry leading pricing and performance (take a look at HeatWave for an example of this). As a cloud advisor, I often see a lack of proper due diligence in cloud adoption decisions. The typical cloud decision is made at the CFO or CIO level without full visibility into the cost of complex application delivery. These are not stupid people. My take is that a deliberate fog is created with many cloud vendors to obfuscate the true cost of business. Three blindsides hit the executives. One, performance at production scale require ala carte additions for reasons outside of the workload's demands. A case in point, a cloud vendor required an 8 vCPU instance to equal the performance of a four core, on-premises box at production workloads. The storage driver ate half of the available CPU and IO traffic was routed over a single vNIC shared with application traffic. This was unexpected by the customer -something hidden in the vendor's implementation. Two, lack of transparency, leads to billing surprises. Another of my clients budgeted $45k for a 3 month production cloud deployment test. The CFO had an anti-pleasant meeting with the CIO when the first month's bill hit $40k. The cause was egress charges. The experiment ended and they stayed on-premises. At least they had tried the workload at production scale. The third is cloud 'divorces' are almost never considered. The cost of 'divorce' is extremely high, cost prohibitive in many cases but not just for egress fees. For instances a client using a cloud native database found initial performance good for inserts, but after time query performance became unacceptable. The promised scaling was not there. Fortunately the lesson was learned while egress was 'affordable' for moving 2TB of production data out. The pain came into play with the additional computing resources needed to export (not copy out) the data out for repatriation. Insult was added by maintaining two production environments for the 10 days to perform the export and required expertise to implement it. That was for one application's analytic database. Now imagine moving full technology stacks for multiple critical application. How would I avoid the issues: 1. If billing isn't fully transparent, then move to another vendor. Too often, a small application gets to a $100k annual spend, the cloud vendor give a fair discount at this point. Once the spend hits a $1m, usually to compensation for vendor deficiencies at production scale, there is no additional assistance offered. The vendor has you exactly where they want you and your are locked. Greater discounts for 3 year terms you say, you've got three years of sub par service and get to spend more money to solution yourself out of the hole. 2. Test at production workloads. Does the spend match what you need or do you need ala carte upgrades to get you where your workload's performance needs to be today and out years? On then other hand are you paying for the architectural deficiency of the vendor compared to your current as built. 3. If you are dissatisfied with the quality of product or more importantly the level of service, can you just leave then and there? A quality cloud vendor will create a great experience pre and post sale. The will not create artificial barriers to departure. The golden rule for a valued cloud vendor is, "I have to prove myself, every day, in every way to keep my customer for tomorrow." If you sense that isn't the demonstrated action of a perspective or current cloud vendor -move along.

Rd6n6 5y ago

Digital oceans bandwidth charges are about 0.01/GB, which is the cheapest I’ve found. Anybody know of similar or better offerings elsewhere?

mritzmann 5y ago
Hetzner Cloud is known in the DACH region (Germany, Austria, Switzerland). They charge 1.19€ ($1) per TB and 20TB per Compute/VM is included.
neop1x 5y ago
Scaleway - egress free

reilly3000 5y ago

In AWS's defense, their egress pricing covers basically 100% of the cost of networking for their whole platform. VPC, ingress, and many other features and services roll up under that bill. On that basis I have to disagree with the analysis in this post. However, their main point is well-made, its a lopsided agreement that traps customer's data where in some cases its just not affordable to leave. Its not well known to customers until they face the bill... there was a story of a giant data project at NASA where they were planning to store hundreds of petabytes of data on AWS, but neglected to account for the costs of egress which through the entire project into peril: https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/ https://www.theregister.com/2020/03/19/nasa_cloud_data_migra...

mark_l_watson 5y ago

I only use AWS because companies I work for are locked in. When I am spending my own money, I love the low costs at Hetzner and when I need to spin up VPSs I find GCP much more convenient to use. I am also evaluating Oracle right now because they have a generous free tier VPS and their services are easy enough to use. I pay $$$ for Apple’s products and services, but I don’t enjoy using AWS. I do enjoy Amazon’s sales and delivery platform for buying physical things, and their digital media, so I don’t mind paying $$ for that.

xtat 5y ago

While it ain't wrong, I'm not sure I want to trade one cartel for another, Cloudflare.

mdavis6890 5y ago

This post is not very fair, and seems to have started with a conclusion "big company screwing their customers," instead of really digging in to learn why things are the way they are for good non-customer-screwing reasons. The post DOES actually call out that wholesale bandwidth is symmetric, but then jumps to the OPPOSITE of the correct interpretation of that: That AWS's pricing to the end customer should be equally symmetric, and the fact that it isn't is evidence of AWS's customer-screwing behavior. The actual reason is that AWS's customers, like those of other cloud providers, are running externally-facing services that mostly send data OUT to their users. Given the inherent symmetry in wholesale bandwidth, it means that AWS has to build out based on their total egress bandwidth. As customers egress more data their users (or wherever/whoever), AWS has to not only pay for more wholesale bandwidth, they also have to build out all the additional infrastructure (routers, etc) to support it. It's very expensive. The ingress bandwidth, being less than the egress bandwidth already noted above, therefore is effectively free to AWS, and they can charge very low rates to their customers for it.