9 ms·
Can someone explain to me why traffic is still so damn expensive with every cloud provider? A while back we managed a site that would serve ~700 TB/mo and paid
by phoboslab 11y ago
Can someone explain to me why traffic is still so damn expensive with every cloud provider?
A while back we managed a site that would serve ~700 TB/mo and paid about $2,000 for the servers in total (SQL, Web servers and caches, including traffic). At Google's $0.08/GB pricing we would've ended up with a whooping $56,000 for the traffic alone. How's that justifiable?
- saganus 11y agoWow, ~700 TB/mo? That does sound like a lot. What kind of site would serve that volume of traffic and not have 56k for operating expenses? I mean, I can think of a few examples like Wikipedia maybe, since they are non-commercial and such, but for a commercial business? Maybe 4chan moves that much without a lot of revenue I would think, or maybe... imgur? but not really sure, I mean, it would seem like they could get that amount easily via ads alone. What was the use case here? Also, I think that 56k for traffic alone kind of depends on context. I mean, how much does Netflix pay for serving their volume of traffic? What I'm saying is, isn't 700 TB a month something that would probably be very expensive no matter the context? Just storing 700TB would cost a lot, no? I'm really curious about your use case here.
- virtuallynathan 11y ago700TB/mo is about 2Gbps - on the open market that should be under $1000/mo. Netflix's total cost is probably below $0.25/Mbps. $56,000/mo would get you over 100Gbps of committed capacity from any major provider (or a mix).
- pyre 11y ago> 700TB/mo is about 2Gbps - on the open market that should be under $1000/mo Is that a fixed-cost sustained pipe though? I was under the impression that (at least at the backbone level) those contracts got more costly the closer to full that your pipe was.
- virtuallynathan 11y agoYes, $1000/mo would get you a 2Gbps commit on a 10Gbps pipe. If you used over 2Gbps (95%ile) for the month, you would pay probably $0.60/Mbps for that excess ($0.10 over the commit price). Some providers dont charge more than the commit price for overage traffic.
- saganus 11y agoInteresting data. Thanks! It's obvious that I had no idea about costs. I honestly thought it would be much more expensive.
- phoboslab 11y agoImage hosting community site - notably without shady popup/layer/scam ads, which probably was the reason for the relatively small income. For a two person team that only worked part time on it, it still made good money. The total dataset was just about 3TB, so storing it was not an issue.
- saganus 11y agoI see. It does make sense. Thanks for satisfying my curiosity :)
- deleted 11y ago[deleted]
- alainv 11y agoWhy is it "not have 56k for operating expenses"? Something that can be had for $2k is not something a healthy business spends $56k on. You should be able to find a better use for those $650k that year.
- superuser2 11y agoThe engineer time to reimplement the other AWS services you're using may be substantially more than the $54k difference in bandwidth costs.
- kijin 11y agoDepends on what they're using AWS for. OP seems to be running a simple and straightforward setup that just happens to use a lot of bandwidth. It doesn't take five full-time engineers to maintain a handful of LEMP servers.
- qaq 11y agoHmm very unlikely. Thats 5 full time people.
- eru 11y agoMore like 2 or 3 if you include overheads. Depends on location, of course.
- vidarh 11y agoNothing prevents you from mixing and matching there are actually AWS services you can't find cheaper equivalents to elsewhere. My experience is that those of my consulting clients that want to migrate off AWS rarely have any problems replacing it. The cost savings usually pay back any development costs and the overall migration effort in 2-3 months at most.
- misiek08 11y agoIf there will be 2 such companies, for that 108k I can wrote services backend that will be comptabile with AWS, so after a year you can transparently switch to that system on your bare metal and sell it to have your services for free. I still didn't make my own "clone", because I can't afford machines to start selling it.
- zifnab06 11y ago1TB/mo is roughly a constant 3mbit/s. So an estimated 2.1gbit/s. I recently had a 1gbit line from he.net quoted at $500 in Seattle.
- airza 11y agoSounds like you should start your own cloud hosting service! I bet you could make a killing.
- johansch 11y agoThat is a key question I have been pondering myself. One theory of mine (perhaps uninformed; I'm not really a networking expert) is that because of the dynamically configurable nature of their systems, they need to use routers rather than relatively dumb and cheap switches at almost every level - in order to have flexible networking and still maintain isolation between customers. This could get quite expensive if you have to pay Cisco/Juniper for this. If this is true Google will have quite an edge with their software defined networking here, I would guess.
- virtuallynathan 11y agoLarge networks like Level3, Cogent, Telia, etc all use big-iron routers (Cisco/Juniper) and will sell you traffic for under $1/Mbps.
- johansch 11y agoThey (Level3, Cogent, Telia) don't have millions of ports though...
- virtuallynathan 11y agoThis is true, but I can't imagine Google/Amazon/Microsoft are using Cisco/Juniper routers at every level of their network.
- stephengillie 11y agoHosts play a big part of SDN in that they support the dvswitches along with guest VMs. Not everything is a Cisco/Juniper. Switching hardware is still common in TOR and egress.
- cenal 11y agoVLAN's and Virtual Appliances in the same environment as the guest machines to facilitate routing should allow for scale without costing these virtualization providers too much.
- mikecb 11y agoTheir CDN interconnect lowers that pricing to ~$0.04/Gb (US).
- virtuallynathan 11y agoThat is still about $13/Mbps, or 26x transit pricing.
- mikecb 11y agoThis isn't wholesale, but at least it's half what they quoted. Additionally, you're only paying to update the assets and CDN fees.
- amazon_not 11y agoThat's still very expensive. Wholesale rates for bandwidth are a fraction of a penny per GB.
- erikpukinskis 11y agoWhen pricing a value-add you want to price it linearly, with a volume discount, but such that after the volume discount the line is still steeper than the base cost curve. That way growing customers feel like they are getting a deal vs small fish, and are incentivized to use as much as they need, but you still drive your margins towards what the market will bear, provided your volume is growing. That curve will eventually squeeze out some of your biggest customers, but you can avoid this by cutting deals for them, e.g. Google with Apple.
- mwfj 11y agoThey are pricing themselves out of the market for traffic-intensive small-fish operations that way though.
- duaneb 11y ago> traffic-intensive small-fish operations Do you have any examples? It seems like it's always been a grow-and-become-profitable-or-die-fast niche.
- g4k 11y agoI run a few websites with video content which leads to 50TB+ per month. The business is profitable, but clearly I would not waste my money on expensive bandwidth.
- simplemath 11y agoGoogle clearly isn't trying to be a porn CDN
- brianwawok 11y agoWhich is funny because through YouTube they have to have the cheapest raw bandwidth in the world. They need two traffic prices.... Fast low latency web traffic for the current 10 cents per GB. Slower more laggy CDN type bandwidth for like 10 cents per TB.
- rs999gti 11y agoHow much do staff salaries and data center rentals add to the cost per server and per GB?
- amazon_not 11y agoWhy do you assume you need a staffed data center to get cheaper bandwidth? Just buy dedicated servers or VPSes, no datacenters or staff needed. The hosting provider takes care of the servers, staff and the datacenter.
- tracker1 11y agoIf I ever got to where my bandwidth fees were even a hundred a month for personal projects, I'd switch over at least part of it to a VPS... As a business, I wouldn't do it until the cost of the bandwidth+hosting exceeded the cost of an extra, dedicated employee to manage the VPS server(s).
- amazon_not 11y agoHow do you figure you need extra, dedicated employees to manage VPSes compared to cloud VMs? The hosting company takes care of the VPS servers, just like Amazon takes care of the AWS servers.
- tracker1 11y agoOnce you need to scale, you need that expertise... if I can use RDS, Azure SQL, or a number of other options to manage database services, or other systems without dedicated staff, that buys time to keep the lights on while actual solutions and features are created... an MVP needs to work... And "wasting" a few hundred a month on hosted services while trying to get something working is better than having to spend that time becoming experts on infrastructure, databases, or any number of other systems. I'm not saying don't optimize, but I am saying that you shouldn't switch infrastructures unless you are saving enough to cover additional talent.
- 11y ago
- baddox 11y agoSeems like the obvious cynical answer is that they do that to encourage you to use more of their services.
- mythz 11y agoTraffic's a luxury tax (along with RAM) that cloud providers assume that big companies can afford to pay if they're getting that much traffic. Outside of the Cloud Providers Traffic is dirt Cheap, Hetzner includes 30TB traffic inclusive in their dedicated server i7 Quad-Core Skylake 64GB DDR4 RAM, 2x250 GB SATA 6 Gb/s SSD for 39 euro /month: https://www.hetzner.de/us/hosting/produkte_rootserver/ex41sssd https://www.hetzner.de/us/hosting/produkte_rootserver/ex41ss... If you don't want to be shaped after you exceed 30TB, Hetzner charges €1.17 per additional TB, so 700TB would come to €783.90 total. Whereas ScaleWay include unlimited traffic in their bare metal servers starting from 12 euro /month: https://blog.scaleway.com/2016/03/08/c2-insanely-affordable-x64-servers/ https://blog.scaleway.com/2016/03/08/c2-insanely-affordable-...
- nitrogen 11y agoHow many people share that Hetzner server for 39euro/month?
- Mo3 11y agoNone.
- annnnd 11y agoActually, one. :)
- sarnowski 11y agoIt's a dedicated bare metal machine for you. The tradeoff with Hetzner is, that it's not expensive Server hardware, so you will encounter hardware problems more often compared to a Dell or HP server.
- mythz 11y agoHetzner also has options for Xeons and Dell PowerEdge servers for a bit more a month, but I've also had great experience with their best value hosting servers, ran a site on it for a couple of years without running into any h/w issues before moving to AWS due to its easy managed RDS, S3, SES services. But if I'd just needed a single dedicated server with great specs I'd use Hetzner in a heartbeat.
- EdHominem 11y ago> How's that justifiable? What, morally?
- bsder 11y ago> Can someone explain to me why traffic is still so damn expensive with every cloud provider? Because The Cloud(tm) IS cheaper--when you start and don't have any real bandwidth or CPU usage. Whereas, every colocation facility I have quoted wants you to commit to a minimum of $500 for some partial cabinet. So, The Cloud(tm) wins the contract and gets to bill in increasing amounts when usage finally goes up. Finally, how many real system administrators still exist who can provision your systems, configure the network, and understand how to connect everything to the network without getting p0wn3d? If you don't have that person, you can't escape The Cloud(tm) even if you wanted to.
- manigandham 11y ago> Finally, how many real system administrators still exist ... a lot? Has there been some shortage of network/infrastructure people lately?
- tracker1 11y agoWell, considering how many small/startup shops expect the developers to also do IT chores, "the cloud" makes the most sense... spending time learning the insides of systems they don't care to truly maintain comes at a cost... time to do other things, or cost to pay someone else to do it. In the end, the cloud makes sense in a lot of scenarios.
- vidarh 11y ago"The cloud" does not mean you don't need real system administrators. I see time and time again companies get bitten by this. Overall devops efforts to run this well on AWS or GCE in my experience tends to be higher than provisioning dedicated systems because you have so many artificial limits imposed on you by the providers that makes things harder. E.g. your example: Understanding how to connect everything to the network without getting hacked is far easier when your private network is physically wired to a separate switch, and your public network is physically behind a firewall and there's no configuration mistake in the world you could do that would change that, so the problem-space to get basic levels of security is reduced to configuring the firewalls correctly. Still plenty of room to shoot yourself in the foot, but in my experience far less so than having people configure their own networking on AWS. As or pricing, yes, if you want to do colo, the initial costs are higher. But dedicated rented servers with monthly contracts are also typically far cheaper than AWS for anything that stays up for more than ~1/3 or so of the time (obviously depends on the hosting povider). If you regularly spin up lots of instances for a short period of time, you should use AWS. But the moment you stop spinning them down again, it's time to rent capacity somewhere else.
- bhouston 11y agoOne popular high traffic site I know build their own CDN to serve the large majority of their data by renting dedicated machines in OVH, Hetzner, etc. I can not remember their actually datacenters for their own CDN but they were not CloudFront or Google Cloud Platform. Supposedly this has saved them immense amounts of money.
- erichocean 11y agoIf your servers are efficient enough (and this is not hard to do these days), it's easy to get bandwidth-limited on a per server basis, i.e. your server could handle more traffic, but you've maxed out the bandwidth available to that particular server. If you can load balance at the client, then you can "talk" to any server at the edge and don't need a router or proxy, so the net result is that you are only paying for whatever bandwidth comes with your OVH (or whatever) boxes. Effectively, you're buying bandwidth and the computer/storage/power/rackspace/etc. that comes with that bandwidth is free. And yeah, it's ridiculously cheaper than AWS or Google's Cloud Platform to do things this way.
- mrmondo 11y agoStorage is also a lot more expensive from 'cloud' providers, people often forget to look at the performance and redundancy and simply look at 'per gb' costs.
- Swannie 11y agoIndeed. The IOPs numbers for the cheaper VMs are not so great. You need IOPs? Suddenly you are paying for a premium instance type. You want replication and/or geo-redudancy with that? Now we're talking $$$ :D
- boulos 11y agoTo clarify, we don't do that on Compute Engine. The number of IOPS you get is tied to the volume size for Persistent Disk. You choose between the two flavors (SSD and regular) and then size your disk. That does mean you have to buy more GiB than you "need" if you want to go faster, but PD is much cheaper than "bigger VM" in most cases. Disclosure: I work on Compute Engine.
- mrmondo 11y agoSo lets say... how much would say a 2TB volume providing a consistent minimum of 100,000 random 4k write IOP/s that's available across multiple VMs at once and must be highly available at say 99.9% cost? * Note: I went to use Compute Engine's cost calculator but it appeared the site was down / under heavy load?
- ChuckMcM 11y agoPerhaps it is like the gas stations that sell gas for $4.99/gal when others sell it for much less. It's only worth their while to sell it if they make a healthy margin so they only sell to people willing to pay that much.
- developer2 11y agoI don't get it. Google says they're going after the big fish in the industry by claiming they have amazing pricing. The servers look good, I'm ready to jump on board. $120-$230 for first TB of egress bandwidth depending on where it goes. No thanks, I can get 2 TB for < $20 elsewhere. These bandwidth costs leave small businesses, and individuals like myself, staying with the smaller competition. I suppose their reasoning is they can chase after that single $400-600 million contract. One major client like that is worth as much as ten million of us little guys paying $50 each. The big cloud providers exist to to serve gigantic enterprises. The rest of us are a drop in bucket and not worth the effort.