4 ms·

Based on the pricing you see at VPS providers, bare metal providers, and colocation, I have long called BS on large cloud provider bandwidth pricing. It's compl

by api 3y ago

Based on the pricing you see at VPS providers, bare metal providers, and colocation, I have long called BS on large cloud provider bandwidth pricing. It's completely insane, like more than 10X higher than VPSes and more than 100-1000X higher than wholesale (bare metal and colo). Take this for example: https://www.fdcservers.com/ https://www.fdcservers.com/ That company is profitable, which means they have to be paying less than this for bandwidth or at least selling bandwidth at cost and paying less for other things. I'm sure they are overselling but even then this model couldn't work if bandwidth were anywhere even close to what cloud charges for it. (We run a few things there for high bandwidth simple services and it works great and seems to be as advertised. There are many other companies with insanely low pricing on bandwidth, but they're one of the cheapest.) The key to understanding why I really think AWS/GCP/etc. price bandwidth the way they do is that ingress is free and egress is absurdly expensive. I call this "roach motel pricing" after the old roach motel ad: "roaches go in, but they don't come out." The obvious objective is to get data to flow effortlessly in but make it costly to either relay data for edge or endpoint first computation or move data out. The whole pricing structure is to encourage cloud providers to suck in the entire computing ecosystem and host it internally within large data centers, which is their business model. It works very well. The instant I saw this pricing structure when cloud first started to be introduced I immediately understood what was happening here.

tyingq 3y ago
Searching for terms like "wholesale bandwidth transit" might be a better reference for comparison of actual cost. The providers you mention are depending somewhat on customers that statistically underutilize what they have and using obscure TOS clauses to chase away heavy users.
- api 3y ago
  That's the list price. You don't pay the list price if you purchase at high volume. Bare metal and VPS providers are surely very high volume customers. This is perhaps an argument against pure DIY colo, but most small scale colo isn't connected "naked" to the Internet but gets racked in a data center that offers a bandwidth package and does peering for you. The only people who peer directly tend to be clouds, bare metal hosts, CDNs, and ISPs. Even if you have your own IP space and an ASN you are usually behind someone who is bulk purchasing transit and handling peering. To be fair very large customers can negotiate costs down at cloud providers too, but you have to be spending tens of thousands a month to even have a chance at getting their attention for any kind of special deal.
  - buggeryorkshire 3y ago
    Company I used to work for had an AWS bill at 7 figures and sometimes 8 (a very time based offering) AFAICT we had a blanket 20% off the bill, nothing special about egress.
  - tyingq 3y ago
    It's probably the closest proxy for which you can get meaningful numbers.
ddorian43 3y ago
> Take this for example: https://www.fdcservers.com/ https://www.fdcservers.com/ You took "as example" literally the cheapest provider that exists in this space. (and lowest network quality/connections)
- api 3y ago
  Okay, then here's more: Hetzner, Datapacket, Hivelocity, Reliablesite, and OVH. I did pick them as an extreme example, but even the more expensive bare metal and colo providers are like 100X cheaper than cloud. VPS providers like Digital Ocean and Vultr are ~10X cheaper. Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet. The asymmetry you usually see on home connections is mostly a DOCSIS (cable modem) limitation or a way of segmenting personal vs. business accounts. BTW we run a few things at FDCServers (among others) and monitor them and they seem fine. We do use multiple providers in case one of them has a huge issue. Edit: these providers can't be overselling too much because the fact that they have such cheap bandwidth makes them a magnet for bandwidth-heavy work loads: game servers, seed boxes, DIY CDNs, TURN servers, etc.
  - marcus0x62 3y ago
    > Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet. Symmetry is the entire problem. 1) to maintain settlement-free peering, you need traffic-symmetry. Amazon — on their own - transmits far more data than they receive. They benefit far more on this account from ingress traffic than more egress. Hence, the economic signal that ingress is free. 2) If you have unbalanced network traffic, your scaling factor becomes the transmit direction that is more heavily utilized. A 100gbps circuit can generally send and receive 100gbps simultaneously. But it can’t be used to send 150gbps and receive 50gbps. So if you have a bunch of people all wanting to transmit from your circuit far more than they receive, you have to upgrade your circuits and routers earlier than you otherwise would for a given aggregate amount of traffic. This is expensive, but adding more ingress traffic is free.
hotnfresh 3y ago
One complicating factor of egress is that it's not all created equal. Some providers have really good peering agreements. Others (even some big players...) have bad ones. Some cheaper-than-AWS options aren't really selling the same thing. It's really hard to evaluate that before you hit issues in production, too.

by api 3y ago

tyingq 3y ago

Searching for terms like "wholesale bandwidth transit" might be a better reference for comparison of actual cost. The providers you mention are depending somewhat on customers that statistically underutilize what they have and using obscure TOS clauses to chase away heavy users.

api 3y ago
That's the list price. You don't pay the list price if you purchase at high volume. Bare metal and VPS providers are surely very high volume customers. This is perhaps an argument against pure DIY colo, but most small scale colo isn't connected "naked" to the Internet but gets racked in a data center that offers a bandwidth package and does peering for you. The only people who peer directly tend to be clouds, bare metal hosts, CDNs, and ISPs. Even if you have your own IP space and an ASN you are usually behind someone who is bulk purchasing transit and handling peering. To be fair very large customers can negotiate costs down at cloud providers too, but you have to be spending tens of thousands a month to even have a chance at getting their attention for any kind of special deal.
- buggeryorkshire 3y ago
  Company I used to work for had an AWS bill at 7 figures and sometimes 8 (a very time based offering) AFAICT we had a blanket 20% off the bill, nothing special about egress.
- tyingq 3y ago
  It's probably the closest proxy for which you can get meaningful numbers.

ddorian43 3y ago

> Take this for example: https://www.fdcservers.com/ https://www.fdcservers.com/ You took "as example" literally the cheapest provider that exists in this space. (and lowest network quality/connections)

api 3y ago
Okay, then here's more: Hetzner, Datapacket, Hivelocity, Reliablesite, and OVH. I did pick them as an extreme example, but even the more expensive bare metal and colo providers are like 100X cheaper than cloud. VPS providers like Digital Ocean and Vultr are ~10X cheaper. Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet. The asymmetry you usually see on home connections is mostly a DOCSIS (cable modem) limitation or a way of segmenting personal vs. business accounts. BTW we run a few things at FDCServers (among others) and monitor them and they seem fine. We do use multiple providers in case one of them has a huge issue. Edit: these providers can't be overselling too much because the fact that they have such cheap bandwidth makes them a magnet for bandwidth-heavy work loads: game servers, seed boxes, DIY CDNs, TURN servers, etc.
- marcus0x62 3y ago
  > Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet. Symmetry is the entire problem. 1) to maintain settlement-free peering, you need traffic-symmetry. Amazon — on their own - transmits far more data than they receive. They benefit far more on this account from ingress traffic than more egress. Hence, the economic signal that ingress is free. 2) If you have unbalanced network traffic, your scaling factor becomes the transmit direction that is more heavily utilized. A 100gbps circuit can generally send and receive 100gbps simultaneously. But it can’t be used to send 150gbps and receive 50gbps. So if you have a bunch of people all wanting to transmit from your circuit far more than they receive, you have to upgrade your circuits and routers earlier than you otherwise would for a given aggregate amount of traffic. This is expensive, but adding more ingress traffic is free.

hotnfresh 3y ago

One complicating factor of egress is that it's not all created equal. Some providers have really good peering agreements. Others (even some big players...) have bad ones. Some cheaper-than-AWS options aren't really selling the same thing. It's really hard to evaluate that before you hit issues in production, too.