4 ms·

there's no way it actually costs them that much right?

kyledrake 3y ago
Not even close. I think a 1gbps ip transit hookup at a data center goes for $200-400 a month right now. And you can push 192TB a month with that. That's not the bulk cost, they buy larger amounts in volume and probably get a huge discount on it. In fact given their size, they might not pay for it at all (peering agreements), so it's just operating costs.
- nixgeek 3y ago
  A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards. Add on the usual and egregious $150-400/mo for a SMF/OS1 crossconnect in many datacenters. Now, I'll grant you, "not even close" and handsome margins likely still applies but "I'm going to hook a server directly into a 1Gbps IP transit port for $200" vs. all the complexity of a hyperscaler network is not apples to apples.
  - amluto 3y ago
    > A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards. Some of that is capex, and some is opex. But it's worth noting that the hyperscalers are doing something far, for more complex than almost any single-tenant physical installation would do. In a hyperscaler cloud, I can run an instance anywhere in the AZ, and I can connect it to my VPC, and I get what appears to be, functions like, and performs as if I'm plugged in to conventional network infrastructure. This is amazing, and the clouds don't even charge for this service as such. There are papers written about how this works, presentations are given, costs are bragged about, and entire industries of software-defined networking are devoted to enabling use cases like this. But an on-prem or normal datacenter installation doesn't need any of this. Your favorite networking equipment vendor will happily sell you a single switch with tens of Tbps of switching bankwidth and individual links that are pushing 1Tbps. It takes maybe 2RU. If you are serving 1 billion DAU, you can serve up a respectable amount of content on that one switch. You could stick all the content in one rack (good luck at that scale), or you can scale across a few racks, and you don't need any of the amazing things that hyperscalers use. And, realistically, very very few companies will ever need to scale past that in a single datacenter -- there aren't a whole lot of places were even 100% market penetration can find 1 billion people all close enough to a single datacenter that it makes sense. So the apples-to-oranges comparison cuts both ways. You really can get a lot of mileage out of just plugging some off-the-shelf hardware into a transit port or two.
    - nixgeek 3y ago
      Notably, anywhere in the AZ is simply a launch constraint, and the hyperscaler can and does place you in the most advantageous way for them which matches against the 'customer promise' (constraint). They in practice do model both the volume and cost of East-West traffic in their network topology and will wherever possible not place Instance A and Instance B all that far apart in the AZ, particularly if that AZ is one of the big ones with 20+ datacenters worth of capacity. All the other points you made are also excellent.
  - kyledrake 3y ago
    The important takeaway message I'm trying to deliver is that AWS is comically overpriced regardless of what they are trying to run. Direct connections aside, reasonable cloud competitors with similar networks are charging $0-1000 for transit and the same amount would cost $8500 on AWS. Cloudflare in particular has demonstrated that you can run these systems so efficiently that they can charge nothing for egress and still make a healthy profit selling related cloud services.

Not even close. I think a 1gbps ip transit hookup at a data center goes for $200-400 a month right now. And you can push 192TB a month with that. That's not the bulk cost, they buy larger amounts in volume and probably get a huge discount on it. In fact given their size, they might not pay for it at all (peering agreements), so it's just operating costs.

nixgeek 3y ago
A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards. Add on the usual and egregious $150-400/mo for a SMF/OS1 crossconnect in many datacenters. Now, I'll grant you, "not even close" and handsome margins likely still applies but "I'm going to hook a server directly into a 1Gbps IP transit port for $200" vs. all the complexity of a hyperscaler network is not apples to apples.
- amluto 3y ago
  > A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards. Some of that is capex, and some is opex. But it's worth noting that the hyperscalers are doing something far, for more complex than almost any single-tenant physical installation would do. In a hyperscaler cloud, I can run an instance anywhere in the AZ, and I can connect it to my VPC, and I get what appears to be, functions like, and performs as if I'm plugged in to conventional network infrastructure. This is amazing, and the clouds don't even charge for this service as such. There are papers written about how this works, presentations are given, costs are bragged about, and entire industries of software-defined networking are devoted to enabling use cases like this. But an on-prem or normal datacenter installation doesn't need any of this. Your favorite networking equipment vendor will happily sell you a single switch with tens of Tbps of switching bankwidth and individual links that are pushing 1Tbps. It takes maybe 2RU. If you are serving 1 billion DAU, you can serve up a respectable amount of content on that one switch. You could stick all the content in one rack (good luck at that scale), or you can scale across a few racks, and you don't need any of the amazing things that hyperscalers use. And, realistically, very very few companies will ever need to scale past that in a single datacenter -- there aren't a whole lot of places were even 100% market penetration can find 1 billion people all close enough to a single datacenter that it makes sense. So the apples-to-oranges comparison cuts both ways. You really can get a lot of mileage out of just plugging some off-the-shelf hardware into a transit port or two.
  - nixgeek 3y ago
    Notably, anywhere in the AZ is simply a launch constraint, and the hyperscaler can and does place you in the most advantageous way for them which matches against the 'customer promise' (constraint). They in practice do model both the volume and cost of East-West traffic in their network topology and will wherever possible not place Instance A and Instance B all that far apart in the AZ, particularly if that AZ is one of the big ones with 20+ datacenters worth of capacity. All the other points you made are also excellent.
- kyledrake 3y ago
  The important takeaway message I'm trying to deliver is that AWS is comically overpriced regardless of what they are trying to run. Direct connections aside, reasonable cloud competitors with similar networks are charging $0-1000 for transit and the same amount would cost $8500 on AWS. Cloudflare in particular has demonstrated that you can run these systems so efficiently that they can charge nothing for egress and still make a healthy profit selling related cloud services.