10 ms·
Having done this computation a number of times, with these machines it would flip over to 'colo' at about 150 machines. The really interesting thing is that ama
by ChuckMcM 11y ago
Having done this computation a number of times, with these machines it would flip over to 'colo' at about 150 machines. The really interesting thing is that amazon costs scale linearly with size while owned infrastructure scale fractionally. So the advantage just keeps growing and growing.
Bottom line, if you can host your app and resources on less than a 150 machines in Amazon you win, more than that and you're leaving money on the table. Once you get to the point where you are deploying new datacenters to support your customers you get a huge boost in operational efficiency.
- ciupicri 11y agoYet Netflix migrated to Amazon and I think that it has way more than 150 machines.
- rhizome 11y agoI have to think they're getting a deal.
- kevcampb 11y agoWhich should be expected when buying in bulk.
- BorisMelnik 11y agothat would be fun to calculate
- vosper 11y agoPlus, they're the number one poster-child for AWS. Netflix running on AWS makes AWS safe for CTOs everywhere. So, yeah, they're definitely getting a deal.
- ChuckMcM 11y agoYes they did. And that is a really great exercise in thinking about what their profitability would be like if they were structured more like YouTube in terms of their server infrastructure. But to the point of many other folks here, if you're all cloud you can really shrink fast if you need too. Lose a millions customers? No problem, just drop some machines. Any business that can price into the monthly customer fee the marginal monthly cost of AWS to support that customer, can keep their costs flat when their customer counts vary. Someone with dedicated hardware is still paying the bills for it, even when their customers leave.
- boomzilla 11y agoI am not sure I'd agree that Netflix could've been more profitable if they went with their own DC. I think Netflix does not utilize that much hardware. Their streaming is on their own CDN, which I heard that are placed directly in the big cable companies' DC. Tracking and storing user profiles and other meta data would not require a huge amount of hardware. I think Netflix also does quite a bit of video encoding and analytics, but these workloads would fit well with EC2.
- ChuckMcM 11y agoThat would be the thing right? Clearly they can compute their marginal cost per subscriber, and they know what the lifetime value of a subscriber is, so they budget what to spend on EC2 infrastructure and movies at Sundance and still meet their net income goals. They would also know how much difference it would make if they needed to add or reduce their EC2 footprint. Sadly I don't expect them to share what those costs are. I have always felt that Amazon could, in theory, always out compete with NetFlix with their Amazon Prime Video because they would only have to charge the marginal rate for the hardware. However to date it seems like the content contracts have kept NetFlix on top.
- abakker 11y agoYes, but a huge reason to go with Amazon is to be able to take money back off the table. Sure, if you have predictably 24/7 load, then you may be better off racking your own at 150 machines. BUT, if you're not sure how much you'll need, and your app is built to scale up and down with demand, then that 150 number starts to slide upwards. Risk mitigation is part of the cost of infrastructure that needs to be factored in, and if your business has a downturn, or your compute needs turn out to be lower, then you can scale down your OPEX pretty quickly. If you had bought them, you'd be stuck with them. (See Zynga) The operating cost of doing it yourself is always hard to pin down, and include many many things. OS/Hypervisor licenses, staff, hardware, cooling, redundancy, disaster recovery, power, or Colo fees etc. The advantage with AWS, is that this just a few numbers on a few various services and it is very predictable.
- boomzilla 11y agoYes, I guess the 150 machine is more like a guideline. It depends very much on your workload. If you need 150 beefy machine running at 90%CPU all the time, then yes, running your colo may come out ahead. On the other hand, if you run a system with an online fleet of say 20 web servers, a couple of beefy databases, and some offline analytic workload, then AWS will definitely come out ahead. And that is not considering other AWS offering. I've come to really like DynamoDB. It takes some time to get used to, but I've found it's solving more and more problems for me, without having to scale OPS engineers. There is a danger of lock-in, but I guess as I am a paying customer, it's not going away anytime soon, and Amazon is probably not jacking up the price either.
- thezilch 11y agoNote: I manage dedicated-server hosting and all web services on Titanfall @ Respawn. Surely your calculation doesn't / can't include bandwidth charges? And if you have Windows hosts... cloud screws you too. You're not wrong that it takes a lot more machines than most shops need, and if you can make your scale elastic, 150 machines probably goes a long way! And to your point, developers undervalue their time a lot, and dealing with colocation and sourcing of hardware and backups can be a real drain to save some on CapEx.
- jpgvm 11y agoExactly, transit cost is the big killer. AWS/GCE/Azure are all really expensive compared to even good multi-homed transit. If you push more than 10TB and you don't need multi-homing you are already ahead. If you do need HA multi-homed transit you are looking more at 50-100TB or so but still, there are a lot of places that easily do that every month, especially if you are doing cross-DC storage replication for example.
- ChuckMcM 11y agoOur transit costs out of our Colo in Santa Clara are $6K/month for two dedicated gigabit lines. That translates roughly to 518 TB of data transfer a month if we could keep the pipes fully lit 24/7. Cogent has offered to make one of our gigabit lines burstable to 10G if we would like, same cost if we don't exceed a monthly average use of 1000Mbps. And we use Linux so no Windows hosts charges. There are more and more tools that are force multipliers for your Site Reliability Engineer (SRE) equivalents. You can't avoid swapping out dead drives, but with the right systems architecture you can make it pretty painless.
- rdl 11y agoWow, you're getting ripped off.
- deleted 11y ago[deleted]
- 11y ago
- Retric 11y agoBulk rates are always going to be unlisted and cheaper. Also, having 150 machines in a single location is rarely optimal.
- kayoone 11y agoStill not convinced about this. I can get a pretty beefy server from Hetzner for 60EUR per month. Skylake 6700, 64 GB RAM, 500 GB Raid1 SSD. One time setup fee of about EUR 100 but can be canceled on months end. On AWS an equally specced machine would run >300 USD per months easily. So it really only makes sense if you can scale your usage up and down a lot during the day. Does not make sense for most of smaller projects. Most of the companies i worked with had an EC2 setup of 3-10 instances that were mostly always up, which is way more expensive than doing the same thing on Hetzner. Basically they just did it because of free AWS credits for startups.
- ChuckMcM 11y agoSlight digression here but "RAID 1 SSD" might not be a good idea, especially if you bought two SSDs at the same time. They are probably from the same lot. My experience with 3,000 SSDs in use is that they seem to fail with a combination of age and rewrites. So I would be concerned that two equivalently aged SSDs with an equivalent set of rewrites (as RAID 1 does) would be quite likely to fail at the same time, or more precisely, when one failed you probably have a statistically harder time recovering the mirror from the other. That is just statistics of course, but one of the things I tried to do in the Blekko infra-structure was to mix the ages of the SSDs to mitigate this risk.
- rbanffy 11y agoThis is good advice even for spinning disks. I once met an unlucky RAID-5 array of three identical disks from the same brand, model, batch and with close serial numbers. When one of them failed, I immediately ordered 3 replacements. They didn't arrive on time. Good thing I had backups.
- matt_wulfeck 11y agoI've heard there's strong correlation in failures by lot number. Can you confirm this? What's the practical way to avoid this? Staggering your SSD purchases?
- hhw 11y agoThis is a false dichotomy. There's no need to flip to colo, when dedicated hosting exists. The provider takes care of the network, hardware, and potentially much more depending on the service agreement. You can even have the provider setup a cloud environment for you, if you want to be able to spin up and down VMs. This eliminates the capex costs, while having much lower opex costs than colo at smaller scale. Considering that you can easily do dedicated for 1/7th the cost of Amazon for the equivalent amount of resources, it doesn't take anywhere near 150 machines to realize cost savings, it's more on the order of one complete physical machine as a dedicated server. You can purchase a 2nd one for redundancy and/or for always available additional capacity, so that you can burst to double your usual peak performance and still save 5/7th the costs of Amazon. If you're expecting to require the ability to scale much higher than 2x, you can also purchase a bunch of servers just for 1 month, or even less time depending on the provider, or take a hybrid approach if you really need an extreme amount of dynamic scalability. The reality though is that such extreme scaling is in the minority of cases. I'm not sure how you arrived at the 150 machines number, but that sounds like 3-4 full cabinets and about right to have a reasonable scale for colo.