3 ms·

At what scale does colocation make sense as opposed to a long-term commitment on AWS or other clouds, where you get a discount?

TechBro8615 6y ago
If you’re building an infrastructure company, or any bandwidth intensive product, it makes sense pretty early. For example, it would be impossible to build a competitive CDN or VPN company based on AWS infrastructure. It would also be hard for Zoom to offer a free tier if they were paying per gigabyte for bandwidth, since it would essentially mean every extra second of a meeting cost them money directly. The fundamental problem is that AWS (or any major cloud) charges you for the amount of “stuff” you put through the pipe ($/gb), but with colocation you can pay a fixed cost for the size of the pipe ($/gbps). This allows you to do your own traffic shaping and absorb bandwidth costs without needing to pass them onto your customers. This is the dirty, open secret of cloud pricing models. It’s also their moat, which makes it infeasible to do something like “build AWS on AWS.”
- kitteh 6y ago
  In China the cloud provider pricing model is that you have a slider bar to select how many Mbps your instance can scale up (maxes at 100 to 200 Mbps). But that has more to do with controlling customers ability to burst whereas most other providers do the GB xferd. Some of the China provides have adopted the GB xferd recently, though.
  - kartickv 6y ago
    So the way the slider works is that you pay for the max mbps irrespective of data transferred?
- kelp 6y ago
  Yeah, this is really where they get you. For context, if you were to buy 10Gbps of dedicated internet transit, he.net is currently advertising that for $900/month. If we convert that to GB per month it's 3,240,000GB, so we can calculate what AWS would charge based on list prices. Using their pricing calculator: https://calculator.aws/#/createCalculator https://calculator.aws/#/createCalculator Outbound from Cloud Front or US West (Oregon) to the Internet: $165,891.11 That's a 184 x increase in price! So yeah, you have to buy networking gear and other stuff, but you can get quite a bit of gear for $165K/month. Now you don't really want to run that 10Gbps link flat out like that, but you get the point. The AWS markup on bandwidth costs is absolutely insane. Pro tip: if you have a large enough cloud provider spend, you can negotiate the bandwidth prices down quite a lot, given their markup, they have some room to move.
donavanm 6y ago
My swag would be around the $500K/mo opex and $5-10M/year capital expenditure mark it’s worth a conversation. I haven’t been deeply involved in infrastructure for a few years now. I did work in compute farms, networking, CDNs, etc for about 15 years previously. In my old comments you can find more detailed math on swagging out actual network & infra costs for “equivalent to cloud” infrastructure.
- kelp 6y ago
  This sounds about right to me. The last time I was doing that kind of work it was at slightly large scale than that, and it always modeled out cheaper to stay with our on gear, in colocation space.
rstuart4133 6y ago
Any scale where you get good utilisation of the metal. You can rent a dedicated 8 core server for $100/mo or so, 1Gbps unlimited usage. If you use it at full throttle it is literally under 1/10'th the price, often under 100'th of the amount you will pay for a cloud host and the thousand little cost addon's. But the trick is you have to actually use it and need it in real time. An AWS instance costs you nothing if you don't use it, and almost nothing if you let them kill it at their whim. Zoom's strategy looks pretty optimal to me. Take the 100 fold price reduction on your predicable load, farm the rest out to the lowest bidder.

TechBro8615 6y ago

If you’re building an infrastructure company, or any bandwidth intensive product, it makes sense pretty early. For example, it would be impossible to build a competitive CDN or VPN company based on AWS infrastructure. It would also be hard for Zoom to offer a free tier if they were paying per gigabyte for bandwidth, since it would essentially mean every extra second of a meeting cost them money directly. The fundamental problem is that AWS (or any major cloud) charges you for the amount of “stuff” you put through the pipe ($/gb), but with colocation you can pay a fixed cost for the size of the pipe ($/gbps). This allows you to do your own traffic shaping and absorb bandwidth costs without needing to pass them onto your customers. This is the dirty, open secret of cloud pricing models. It’s also their moat, which makes it infeasible to do something like “build AWS on AWS.”

kitteh 6y ago
In China the cloud provider pricing model is that you have a slider bar to select how many Mbps your instance can scale up (maxes at 100 to 200 Mbps). But that has more to do with controlling customers ability to burst whereas most other providers do the GB xferd. Some of the China provides have adopted the GB xferd recently, though.
- kartickv 6y ago
  So the way the slider works is that you pay for the max mbps irrespective of data transferred?
kelp 6y ago
Yeah, this is really where they get you. For context, if you were to buy 10Gbps of dedicated internet transit, he.net is currently advertising that for $900/month. If we convert that to GB per month it's 3,240,000GB, so we can calculate what AWS would charge based on list prices. Using their pricing calculator: https://calculator.aws/#/createCalculator https://calculator.aws/#/createCalculator Outbound from Cloud Front or US West (Oregon) to the Internet: $165,891.11 That's a 184 x increase in price! So yeah, you have to buy networking gear and other stuff, but you can get quite a bit of gear for $165K/month. Now you don't really want to run that 10Gbps link flat out like that, but you get the point. The AWS markup on bandwidth costs is absolutely insane. Pro tip: if you have a large enough cloud provider spend, you can negotiate the bandwidth prices down quite a lot, given their markup, they have some room to move.

donavanm 6y ago

My swag would be around the $500K/mo opex and $5-10M/year capital expenditure mark it’s worth a conversation. I haven’t been deeply involved in infrastructure for a few years now. I did work in compute farms, networking, CDNs, etc for about 15 years previously. In my old comments you can find more detailed math on swagging out actual network & infra costs for “equivalent to cloud” infrastructure.

kelp 6y ago
This sounds about right to me. The last time I was doing that kind of work it was at slightly large scale than that, and it always modeled out cheaper to stay with our on gear, in colocation space.

rstuart4133 6y ago

Any scale where you get good utilisation of the metal. You can rent a dedicated 8 core server for $100/mo or so, 1Gbps unlimited usage. If you use it at full throttle it is literally under 1/10'th the price, often under 100'th of the amount you will pay for a cloud host and the thousand little cost addon's. But the trick is you have to actually use it and need it in real time. An AWS instance costs you nothing if you don't use it, and almost nothing if you let them kill it at their whim. Zoom's strategy looks pretty optimal to me. Take the 100 fold price reduction on your predicable load, farm the rest out to the lowest bidder.