10 ms·

I don't understand how it is possible in a functioning market economy to have 8000% markup on a utility product like bandwidth. Seems more like an oligopoly whe

by danols 5y ago

I don't understand how it is possible in a functioning market economy to have 8000% markup on a utility product like bandwidth. Seems more like an oligopoly where a few big actors with a big moat has agreed to keep prices inflated. How can a sustained 8000% markup for a product like bandwidth not be considered price fixing? source: https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/

spoonjim 5y ago
AWS is a luxury product and the outbound charge is the delivery fee. You will also find that the delivery fee for a Louis Vuitton bag is more than for a pizza.
- tw04 5y ago
  AWS spent the first decade of existence advertising to executives as a way to significantly decrease their cost of IT, not as the Louis Vuitton of IT.
  - spoonjim 5y ago
    Even a Rolls Royce is cheaper than building your own equivalent car.
    - jleahy 5y ago
      If you need one car, but probably not if you need tens or hundreds of thousands.
- jollybean 5y ago
  No, it's a very valuable tool for corporate IT. The egress is a way to snare a lot of extra margin. It's 100% clear that orgs. wanting to host large public platforms will have 'major concerns' over this pricing issue but that's not their target market. I'll bet most HNeers are thinking in terms of 'Hosting my App' there vs. AWS Bread and Butter is mostly hosting corporate IT services, which is a different thing.
missedthecue 5y ago
Because bandwidth isn't the only AWS product, but you can't buy bandwidth for cheap from XYZ Inc. and connect that to your AWS account. That's not technically possible, you have to buy it from AWS. So instead of using the cheapest vendor, companies use AWS for a ton of reasons (broad support, everyone knows how to work with it so it makes recruiting easier, hundreds of features, etc...) and their high bandwidth bill is worth it.
Aicy 5y ago
This is like going to a supermarket and complaining that the plastic bags for you to store your shopping in cost 10 cents which is a 8000% markup, as if plastic bags are why you're in a supermarket
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- georgyo 5y ago
  This is a horrible example that doesn't not apply cleanly. To fix it, let's change it from buy per bag to per gram you want to leave the store. So your canned soup costs at least 3 times more to leave the supermarket then in cost to buy it than to eat it in the supermarket. This is not even an exaggeration. s3 is 0.024/GB and egress bandwidth is 0.09/GB
slownews45 5y ago
1) $50/TB is not insane pricing relatively. I used to pay something like $3/GB (way back) which would be $3,000/TB. There are still plenty of folks charging high rates. 2) $50/TB may not be a big part of some customers bills. If it is on cloudfront you drop to 0.02/GB or $20/TB for larger volumes. 3) AWS doesn't charge separately for some services that wrap around networking. Their nitro instances have a pretty good networking story. 4) Especially with this new free tier on cloudfront, a fair number of users will never hit 1TB per month (free) data.
- api 5y ago
  It’s absolutely insane compared with what bare metal providers charge.
  - VWWHFSfQ 5y ago
    what do bare metal providers charge
    - teraflop 5y ago
      Just as an example: Hetzner's dedicated servers, which start around US$30/month, come with unlimited 1gbit connections (really unlimited, not "unlimited until we decide to throttle you"). If you max out that connection non-stop, you can push about 330 TB/month. The same amount of bandwidth from EC2 would cost roughly US$20,000.
      - FpUser 5y ago
        Dedicated servers from Hetzner and OVH is what I use. Very sweet deals they have.
      - spenczar5 5y ago
        Are you actually able to max that out to all network peers? Have you measured this? That sounds fantastic.
bluedino 5y ago
How much margin do you think there is in coffee or soft drinks?
- rad_gruchalski 5y ago
  Apparently nowhere near the 8000%. After https://www.profitableventure.com/income-soda-retailers-margin-bottle/ https://www.profitableventure.com/income-soda-retailers-marg...: > Restaurants and bars have around a 70% profit margin on a bottle of soda and soft drink, while retailers typically have between 30–50%.
  - adventured 5y ago
    Coke and Pepsi have that kind of mark-up on their aluminum can sugar water products.
  - jollybean 5y ago
    First - that is 'huge margin' far beyond their other offerings, but on fountain soda it's much greater. Theatres make 0% margin on the Ticket and probably 90% on the soda and popocorn. The analogy is reasonable: 'The Food' , 'The Film' or 'The EC2's are the primary ticket item and it's ballpark competitive on a cost basis. AWS is not competing to be the cheap, low-margin host for YourFreeImageSite.Com and so their prices are going to be different on that point.
  - bluedino 5y ago
    What about a $3 Coke in a vending machine? At a grocery store: 24 pack of Coke being $7 but one 20oz bottle being 1.99? Or that $6 coke at the movies, ballpark, or Disneyland?
    - rad_gruchalski 5y ago
      $7 for 24 cans of Coke? I live in a wrong country.
netwo233gur 5y ago
The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of all of the magic that happens inside AWS between those two things. I've seen some of the inner workings of the big cloud providers' networking stacks. The networking infrastructure, the software that runs it, the software that exposes it to customers, the thousands of engineers working at any given moment in AWS/GCP/Azure's NOCs to maintain uptime are truly some of the most impressive technical marvels I have ever seen. They aren't as sexy to discuss on HN as something like the managed containers services, functions as a service, EC2 etc, but the networking stacks like the VPC, NAT gateways, subnet routing, privatelinks, security groups, ENIs, nitro cards, etc are pure magic as far as I'm concerned and are so so so much more complicated than a standard data center's networking stack, or even Cloudflare's stack. To use Cloudflare's "bucket of water" metaphor, AWS isn't even close to just being a dumb bucket of water that you fill with water and then get charged to take out the water. There is so much that happens inside of that bucket to segment your water into different pipes, routing your water in all kinds of customer-customizable ways for many different use cases, mixing/heating/cooling your water as you need, all while guaranteeing things like making sure your water arrives exactly where it is supposed to arrive and doesn't get contaminated or leaked along the way. Does AWS make a big markup on bandwidth? Yea, surely they do. But is it as simple as Cloudflare says it is? Not even close.
- VectorLock 5y ago
  It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.
  - runlevel1 5y ago
    S3 has its own data retrieval costs, as do several of their managed services. Those are separate charges from the EC2 costs the Cloudflare blog post discussed.
  - Guvante 5y ago
    Egress does not mean S3 or Cloudflare. Egress is the raw pipe billing from AWS to the wider internet. Other services are priced differently.
jollybean 5y ago
TOC 'Total Cost of Ownership'. Nobody is paying for egress, they are paying for everything else. AWS is focused on corporate IT so egress was always a secondary thing for them. It might actually be more profitable for them to have uber-markups on that egress then to serve the web space that requires cheap bandwidth.
VectorLock 5y ago
Its not 8000% on a markup on raw metered bandwidth, they're serving data from a service they operate. Comparing acting like they're just "marking up" bandwidth is an unfair comparison.
throwaway984393 5y ago
What do you mean, how is it possible? That's how markets work. You open a booth in the market, and you can ask any price you want. You can ask for a nickle or a million bucks. There's no rule about how much money you can ask for. As for your assertion that there's a price fixing conspiracy, who do you imagine they're colluding with? What other company do you find charging these prices?
- adventured 5y ago
  The parent comment misunderstands that most of Amazon's customers are very clearly not extremely concerned about the bandwidth costs (even if they'd all happily receive a lower bandwidth bill from AWS). They're not there for the cheap bandwidth. That's the central flaw in the parent's bafflement about markets, they did a exceptionally flat appraisal of the AWS value proposition, as though AWS isn't a gigantic business of many dozens of service offerings which act as a customer magnet and retention mechanism.
  - whoknowswhat11 5y ago
    And many customers simply are not doing petabytes per month. AWS wasn't meant to support Netflix type loads, those guys will build their own CDN's. For everyone else who wants security groups (totally for free) when some other firewall vendors would charge a small fortune to provision 1Gbps capacity for these types of services... they are fine with the price. The other issue cloudflare doesn't understand is the bandwidth pricing they quote is for capacity (ie, 10Gbps). AWS has to have enough capacity to serve the peak, but the customer only has to pay for data used. I'm sure at low points the data is free, but maintaining the black friday / superbowl capacity is expensive, so you are paying a premium for that too.
    - tomByrer 5y ago
      Urm, Netflix is BUILT ON AWS (or at least was 5yr ago): > Netflix uses Amazon Web Services (AWS) for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS. https://aws.amazon.com/solutions/case-studies/netflix-kinesis-data-streams/ https://aws.amazon.com/solutions/case-studies/netflix-kinesi...
amluto 5y ago
Because AWS doesn’t actually want to collect the egress fees. They want you to avoid egress entirely for high-value services, which has all kinds of excellent implications for AWS: Third party SaaS offerings that move large amounts of data are effectively forced to host in AWS. Want a small number of high-value servers (e.g. big GPUs, etc) in your own data center or colo to use for non-availability-critical purposes integrated with the rest of your AWS stack? You’d better price in egress! Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.
- deleted 5y ago
  [deleted]
- colordrops 5y ago
  That still sounds like price fixing, but for a different purpose.
- StopHammoTime 5y ago
  amluto is spot on. AWS don't want to make money off of egress, they want to make it non-viable to move anything out of AWS that isn't being served to an end consumer.
  - _hcuq 5y ago
    Time to move hosting back in house.
  - ignoramous 5y ago
    It is because amulo has a point is why Cloudflare stands a good chance to disrupt AWS in at least few market segments, if not all. More: https://stratechery.com/2021/cloudflares-disruption/ https://stratechery.com/2021/cloudflares-disruption/
- torginus 5y ago
  I wonder, if you made a video streaming app on top of AWS you could negotiate a significant reduction of fees, since you're not ruining their business by transferring out valuable data to competitors. And if that's the case, I wonder if that constitutes a breach of net neutrality in practice, since in practice, a major part of bandwidth costs will depend on what kind of information you are transferring.
adam_arthur 5y ago
The thing about a competitive market is that it requires that cost of switching is low. The higher the cost of switching, the more leverage the company has over you. SaaS and cloud providers in particular have a lot of leverage over their customers due to the typically high cost of switching. E.g. what does it cost to move your whole infrastructure to a new cloud? They can basically gouge you up to that cost threshold. These cost of switching actually leads to a market that is not competitive in the traditional sense. Yes, competition on the surface, but monopoly within. It's true that gouging can't be as egregious as a true monopoly environment, but certainly much higher than a low cost of switching environment. The big tell is margins. Anytime a company can sustain excessively high margins, it's usually a tell that theres a lack of a competitive market. A lot of software companies get high margins either by being first to market (by many years), or leveraging high cost of switching. Despite software being cheap to deliver, it's also cheap for your competitors to deliver, so margins should be low in a highly competitive market, regardless of marginal cost of production. I do believe new regulation is needed to handle these business models such that we can continue to foster a competitive environment. It would have to be very carefully crafted to prevent unintended second order effects of course.
butMyside 5y ago
So it is: https://www.nytimes.com/2020/10/25/technology/apple-google-search-antitrust.html https://www.nytimes.com/2020/10/25/technology/apple-google-s... I mean the politicians are in on it, and turn against them when their ability to control elections for their own purposes comes into question. Look at the “testimony” of Robinhood guy. It’s a scripted TV show; you lose.

by danols 5y ago

spoonjim 5y ago

AWS is a luxury product and the outbound charge is the delivery fee. You will also find that the delivery fee for a Louis Vuitton bag is more than for a pizza.

tw04 5y ago
AWS spent the first decade of existence advertising to executives as a way to significantly decrease their cost of IT, not as the Louis Vuitton of IT.
- spoonjim 5y ago
  Even a Rolls Royce is cheaper than building your own equivalent car.
  - jleahy 5y ago
    If you need one car, but probably not if you need tens or hundreds of thousands.
jollybean 5y ago
No, it's a very valuable tool for corporate IT. The egress is a way to snare a lot of extra margin. It's 100% clear that orgs. wanting to host large public platforms will have 'major concerns' over this pricing issue but that's not their target market. I'll bet most HNeers are thinking in terms of 'Hosting my App' there vs. AWS Bread and Butter is mostly hosting corporate IT services, which is a different thing.

missedthecue 5y ago

Because bandwidth isn't the only AWS product, but you can't buy bandwidth for cheap from XYZ Inc. and connect that to your AWS account. That's not technically possible, you have to buy it from AWS. So instead of using the cheapest vendor, companies use AWS for a ton of reasons (broad support, everyone knows how to work with it so it makes recruiting easier, hundreds of features, etc...) and their high bandwidth bill is worth it.

Aicy 5y ago

This is like going to a supermarket and complaining that the plastic bags for you to store your shopping in cost 10 cents which is a 8000% markup, as if plastic bags are why you're in a supermarket

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
georgyo 5y ago
This is a horrible example that doesn't not apply cleanly. To fix it, let's change it from buy per bag to per gram you want to leave the store. So your canned soup costs at least 3 times more to leave the supermarket then in cost to buy it than to eat it in the supermarket. This is not even an exaggeration. s3 is 0.024/GB and egress bandwidth is 0.09/GB

slownews45 5y ago

1) $50/TB is not insane pricing relatively. I used to pay something like $3/GB (way back) which would be $3,000/TB. There are still plenty of folks charging high rates. 2) $50/TB may not be a big part of some customers bills. If it is on cloudfront you drop to 0.02/GB or $20/TB for larger volumes. 3) AWS doesn't charge separately for some services that wrap around networking. Their nitro instances have a pretty good networking story. 4) Especially with this new free tier on cloudfront, a fair number of users will never hit 1TB per month (free) data.

api 5y ago
It’s absolutely insane compared with what bare metal providers charge.
- VWWHFSfQ 5y ago
  what do bare metal providers charge
  - teraflop 5y ago
    Just as an example: Hetzner's dedicated servers, which start around US$30/month, come with unlimited 1gbit connections (really unlimited, not "unlimited until we decide to throttle you"). If you max out that connection non-stop, you can push about 330 TB/month. The same amount of bandwidth from EC2 would cost roughly US$20,000.
    - FpUser 5y ago
      Dedicated servers from Hetzner and OVH is what I use. Very sweet deals they have.
    - spenczar5 5y ago
      Are you actually able to max that out to all network peers? Have you measured this? That sounds fantastic.

bluedino 5y ago

How much margin do you think there is in coffee or soft drinks?

rad_gruchalski 5y ago
Apparently nowhere near the 8000%. After https://www.profitableventure.com/income-soda-retailers-margin-bottle/ https://www.profitableventure.com/income-soda-retailers-marg...: > Restaurants and bars have around a 70% profit margin on a bottle of soda and soft drink, while retailers typically have between 30–50%.
- adventured 5y ago
  Coke and Pepsi have that kind of mark-up on their aluminum can sugar water products.
- jollybean 5y ago
  First - that is 'huge margin' far beyond their other offerings, but on fountain soda it's much greater. Theatres make 0% margin on the Ticket and probably 90% on the soda and popocorn. The analogy is reasonable: 'The Food' , 'The Film' or 'The EC2's are the primary ticket item and it's ballpark competitive on a cost basis. AWS is not competing to be the cheap, low-margin host for YourFreeImageSite.Com and so their prices are going to be different on that point.
- bluedino 5y ago
  What about a $3 Coke in a vending machine? At a grocery store: 24 pack of Coke being $7 but one 20oz bottle being 1.99? Or that $6 coke at the movies, ballpark, or Disneyland?
  - rad_gruchalski 5y ago
    $7 for 24 cans of Coke? I live in a wrong country.

netwo233gur 5y ago

The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of all of the magic that happens inside AWS between those two things. I've seen some of the inner workings of the big cloud providers' networking stacks. The networking infrastructure, the software that runs it, the software that exposes it to customers, the thousands of engineers working at any given moment in AWS/GCP/Azure's NOCs to maintain uptime are truly some of the most impressive technical marvels I have ever seen. They aren't as sexy to discuss on HN as something like the managed containers services, functions as a service, EC2 etc, but the networking stacks like the VPC, NAT gateways, subnet routing, privatelinks, security groups, ENIs, nitro cards, etc are pure magic as far as I'm concerned and are so so so much more complicated than a standard data center's networking stack, or even Cloudflare's stack. To use Cloudflare's "bucket of water" metaphor, AWS isn't even close to just being a dumb bucket of water that you fill with water and then get charged to take out the water. There is so much that happens inside of that bucket to segment your water into different pipes, routing your water in all kinds of customer-customizable ways for many different use cases, mixing/heating/cooling your water as you need, all while guaranteeing things like making sure your water arrives exactly where it is supposed to arrive and doesn't get contaminated or leaked along the way. Does AWS make a big markup on bandwidth? Yea, surely they do. But is it as simple as Cloudflare says it is? Not even close.

VectorLock 5y ago
It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.
- runlevel1 5y ago
  S3 has its own data retrieval costs, as do several of their managed services. Those are separate charges from the EC2 costs the Cloudflare blog post discussed.
- Guvante 5y ago
  Egress does not mean S3 or Cloudflare. Egress is the raw pipe billing from AWS to the wider internet. Other services are priced differently.

jollybean 5y ago

TOC 'Total Cost of Ownership'. Nobody is paying for egress, they are paying for everything else. AWS is focused on corporate IT so egress was always a secondary thing for them. It might actually be more profitable for them to have uber-markups on that egress then to serve the web space that requires cheap bandwidth.

VectorLock 5y ago

Its not 8000% on a markup on raw metered bandwidth, they're serving data from a service they operate. Comparing acting like they're just "marking up" bandwidth is an unfair comparison.

throwaway984393 5y ago

What do you mean, how is it possible? That's how markets work. You open a booth in the market, and you can ask any price you want. You can ask for a nickle or a million bucks. There's no rule about how much money you can ask for. As for your assertion that there's a price fixing conspiracy, who do you imagine they're colluding with? What other company do you find charging these prices?

adventured 5y ago
The parent comment misunderstands that most of Amazon's customers are very clearly not extremely concerned about the bandwidth costs (even if they'd all happily receive a lower bandwidth bill from AWS). They're not there for the cheap bandwidth. That's the central flaw in the parent's bafflement about markets, they did a exceptionally flat appraisal of the AWS value proposition, as though AWS isn't a gigantic business of many dozens of service offerings which act as a customer magnet and retention mechanism.
- whoknowswhat11 5y ago
  And many customers simply are not doing petabytes per month. AWS wasn't meant to support Netflix type loads, those guys will build their own CDN's. For everyone else who wants security groups (totally for free) when some other firewall vendors would charge a small fortune to provision 1Gbps capacity for these types of services... they are fine with the price. The other issue cloudflare doesn't understand is the bandwidth pricing they quote is for capacity (ie, 10Gbps). AWS has to have enough capacity to serve the peak, but the customer only has to pay for data used. I'm sure at low points the data is free, but maintaining the black friday / superbowl capacity is expensive, so you are paying a premium for that too.
  - tomByrer 5y ago
    Urm, Netflix is BUILT ON AWS (or at least was 5yr ago): > Netflix uses Amazon Web Services (AWS) for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS. https://aws.amazon.com/solutions/case-studies/netflix-kinesis-data-streams/ https://aws.amazon.com/solutions/case-studies/netflix-kinesi...

amluto 5y ago

Because AWS doesn’t actually want to collect the egress fees. They want you to avoid egress entirely for high-value services, which has all kinds of excellent implications for AWS: Third party SaaS offerings that move large amounts of data are effectively forced to host in AWS. Want a small number of high-value servers (e.g. big GPUs, etc) in your own data center or colo to use for non-availability-critical purposes integrated with the rest of your AWS stack? You’d better price in egress! Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.

deleted 5y ago
[deleted]
colordrops 5y ago
That still sounds like price fixing, but for a different purpose.
StopHammoTime 5y ago
amluto is spot on. AWS don't want to make money off of egress, they want to make it non-viable to move anything out of AWS that isn't being served to an end consumer.
- _hcuq 5y ago
  Time to move hosting back in house.
- ignoramous 5y ago
  It is because amulo has a point is why Cloudflare stands a good chance to disrupt AWS in at least few market segments, if not all. More: https://stratechery.com/2021/cloudflares-disruption/ https://stratechery.com/2021/cloudflares-disruption/
torginus 5y ago
I wonder, if you made a video streaming app on top of AWS you could negotiate a significant reduction of fees, since you're not ruining their business by transferring out valuable data to competitors. And if that's the case, I wonder if that constitutes a breach of net neutrality in practice, since in practice, a major part of bandwidth costs will depend on what kind of information you are transferring.

adam_arthur 5y ago

The thing about a competitive market is that it requires that cost of switching is low. The higher the cost of switching, the more leverage the company has over you. SaaS and cloud providers in particular have a lot of leverage over their customers due to the typically high cost of switching. E.g. what does it cost to move your whole infrastructure to a new cloud? They can basically gouge you up to that cost threshold. These cost of switching actually leads to a market that is not competitive in the traditional sense. Yes, competition on the surface, but monopoly within. It's true that gouging can't be as egregious as a true monopoly environment, but certainly much higher than a low cost of switching environment. The big tell is margins. Anytime a company can sustain excessively high margins, it's usually a tell that theres a lack of a competitive market. A lot of software companies get high margins either by being first to market (by many years), or leveraging high cost of switching. Despite software being cheap to deliver, it's also cheap for your competitors to deliver, so margins should be low in a highly competitive market, regardless of marginal cost of production. I do believe new regulation is needed to handle these business models such that we can continue to foster a competitive environment. It would have to be very carefully crafted to prevent unintended second order effects of course.

butMyside 5y ago

So it is: https://www.nytimes.com/2020/10/25/technology/apple-google-search-antitrust.html https://www.nytimes.com/2020/10/25/technology/apple-google-s... I mean the politicians are in on it, and turn against them when their ability to control elections for their own purposes comes into question. Look at the “testimony” of Robinhood guy. It’s a scripted TV show; you lose.