4 ms·

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

by nik736 10y ago

asptimothy 10y ago
I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter. My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's migration off AWS (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...).
- GABeech 10y ago
  That's a big giant "it depends." It depends on what your service looks like: CPU intensive, Memory Intensive, Storage intensive? (In reality some unique mix). You probably won't see a huge savings year one, as you'll be spinning up a lot of new things and have a fairly large CapEx expenditure. Now if your growth pattern is steady/predictable then you should be able to plan out your hardware buys or do a hybrid solution to handle traffic bursts. One of the nice things about running your own hardware is that there are some costs that are easier to control. Don't need new hardware? Don't need to spend on new hardware for example. You also have much more control over your environment so you are able to really optimize your code, and infrastructure so that you don't need to scale as large system wise. But, back to the question on how to model it? You just gotta dig in, and make some educated guesses about performance,test and repeat.
- user5994461 10y ago
  It's rather simple, you count the days you are wasting on handling the low level hardware. The entry cost is £500 per day or $1000 for an engineer. 5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week. These things would have been 5 minutes on a cloud.
- QuinnyPig 10y ago
  It depends upon your constraints. As a for-instance, I was asked this exact question a year or so ago about an on-prem object store instead of S3. The break-even price to go multi-region was ~15PB on paper; that included DC space, hardware, software (build vs buy was another factored discussion), and the staff to run it. That assessment was delivered with the caveat that their uptime was not going to approach S3's by any stretch of the imagination-- and their infrastructure outages weren't going to sync with "most of the internet's." It's a complex topic, and there are many hidden costs...
rbranson 10y ago
It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows down the decision-making process and seriously complicates experimentation. The number of people who know how to build out datacenters at the scale of thousands of machines is vanishingly small. We'd also need to replace IaaS services like ECS, ELB, ElastiCache, RDS, and DynamoDB. There are certainly off-the-shelf replacements, but we'd need to build-out the expertise within our teams to operate these systems. We're talking roughly a dozen or so engineers working full time for many, many months to get these systems in place from scratch, on top of the even larger effort to design and build out datacenters. I'd much rather plow those cycles into efforts like expanding to multiple regions and improving reliability of internal services. That's a much better return on investment for our customers. Right now we're in the sweet spot for the cloud. We're way too big to run on trivial amounts of hardware, growing at a rate that makes it difficult to stay ahead of demand in a datacenter-centric world, and too small to justify investing in a scalable hardware build-out.
- user5994461 10y ago
  Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more. Disclaimer: I am NOT affiliated to Google in any way.
- QuinnyPig 10y ago
  > It could be if our workload was relatively stable +1. If your entire environment is steady-state, it may be worth considering a migration to bare metal. That said, you can expect: * Worse uptime * A fair bit of retraining / hiring different skillsets * A loss of engineering focus as they work on the migration instead of feature expansion It can make sense economically, but that discussion goes well past "the monthly AWS bill."
- lafay 10y ago
  This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistakes-your-company-must-not-make/ http://firstround.com/review/the-three-infrastructure-mistak...

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

by nik736 10y ago

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

asptimothy 10y ago

I was curious about that myself. What's the best way to model the costs associated with maintenance overhead that comes with bare-metal, vs the savings from managing bare-metal servers using a service (like packet), co-locating your own hardware or running your own datacenter. My gut feeling is that you have to get to an extraordinary size to realize any meaningful savings, but that's primarily based on Dropbox's migration off AWS (https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...).

GABeech 10y ago
That's a big giant "it depends." It depends on what your service looks like: CPU intensive, Memory Intensive, Storage intensive? (In reality some unique mix). You probably won't see a huge savings year one, as you'll be spinning up a lot of new things and have a fairly large CapEx expenditure. Now if your growth pattern is steady/predictable then you should be able to plan out your hardware buys or do a hybrid solution to handle traffic bursts. One of the nice things about running your own hardware is that there are some costs that are easier to control. Don't need new hardware? Don't need to spend on new hardware for example. You also have much more control over your environment so you are able to really optimize your code, and infrastructure so that you don't need to scale as large system wise. But, back to the question on how to model it? You just gotta dig in, and make some educated guesses about performance,test and repeat.
user5994461 10y ago
It's rather simple, you count the days you are wasting on handling the low level hardware. The entry cost is £500 per day or $1000 for an engineer. 5 hours on the phone to find the hardware and agree on the order with DELL + an afternoon with customer support because they shipped the servers without hard drives + your project is delayed by an entire week because you don't have the resources => 1 day + 1 day + half a week. These things would have been 5 minutes on a cloud.
QuinnyPig 10y ago
It depends upon your constraints. As a for-instance, I was asked this exact question a year or so ago about an on-prem object store instead of S3. The break-even price to go multi-region was ~15PB on paper; that included DC space, hardware, software (build vs buy was another factored discussion), and the staff to run it. That assessment was delivered with the caveat that their uptime was not going to approach S3's by any stretch of the imagination-- and their infrastructure outages weren't going to sync with "most of the internet's." It's a complex topic, and there are many hidden costs...

rbranson 10y ago

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows down the decision-making process and seriously complicates experimentation. The number of people who know how to build out datacenters at the scale of thousands of machines is vanishingly small. We'd also need to replace IaaS services like ECS, ELB, ElastiCache, RDS, and DynamoDB. There are certainly off-the-shelf replacements, but we'd need to build-out the expertise within our teams to operate these systems. We're talking roughly a dozen or so engineers working full time for many, many months to get these systems in place from scratch, on top of the even larger effort to design and build out datacenters. I'd much rather plow those cycles into efforts like expanding to multiple regions and improving reliability of internal services. That's a much better return on investment for our customers. Right now we're in the sweet spot for the cloud. We're way too big to run on trivial amounts of hardware, growing at a rate that makes it difficult to stay ahead of demand in a datacenter-centric world, and too small to justify investing in a scalable hardware build-out.

user5994461 10y ago
Try all the same on Google Cloud, you'll save half the money compared to AWS, probably more. Disclaimer: I am NOT affiliated to Google in any way.
QuinnyPig 10y ago
> It could be if our workload was relatively stable +1. If your entire environment is steady-state, it may be worth considering a migration to bare metal. That said, you can expect: * Worse uptime * A fair bit of retraining / hiring different skillsets * A loss of engineering focus as they work on the migration instead of feature expansion It can make sense economically, but that discussion goes well past "the monthly AWS bill."
lafay 10y ago
This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure: http://firstround.com/review/the-three-infrastructure-mistakes-your-company-must-not-make/ http://firstround.com/review/the-three-infrastructure-mistak...