8 ms·

Honest question: when you get anywhere near six figure monthly bills, isn't it the time to migrate to your own hardware? Sure, there are several pros and cons

Honest question: when you get anywhere near six figure monthly bills, isn't it the time to migrate to your own hardware? Sure, there are several pros and cons to weigh, but if the application isn't locked in, a migration to metal could make sense. Anyone went through this?

djb_hackernews 9y ago
Depends on your workload. If you have a constant workload, then sure. But if you have variable workloads then AWS/cloud providers are a no brainer. Source: 6 figure/mo AWS spend. We use pretty much every infrastructure piece in AWS in multiple regions and to replicate that flexibility in our own data centers would be an incredible cost.
- mattbillenstein 9y ago
  Your workload doesn't need to be super constant either -- I think I ran the numbers some time ago, but for larger boxes, if you needed the machines for more than 8 hours a day, it was cheaper to run them in a DC and just leave them idle the rest of the time than to run them for that 8+ hours on AWS and shut them down for the rest of the time.
kevinburke 9y ago
At the very least you'd need people with experience running bare metal, delay product pipeline to work on the transition, train the team, build replacements for all of the AWS tools, rebuild this monitoring pipeline without Cloudwatch. Probably looking at $500k in engineering and staffing and training costs at the very least, just to get back to where you started, not to mention the progress you didn't make on the product.
bdcravens 9y ago
I would imagine that unless you have staff sitting on their hands, that level of hardware commitment would require at least one extra employee, generally at a cost of 6 figures a year.
- Aeolun 9y ago
  Not 6 figures a month though.
  - bdcravens 9y ago
    Never said it was an equal cost. Merely pointing out that overhead expenses will cut into cost savings. Your cloud spend isn't just cost of hardware + cloud provider's profit margin.
- bsenftner 9y ago
  Nope, these days servers pretty much run themselves. One just needs to make sure hardware is not failing, and that is pretty much it. Really. Speaking from experience.
  - bdcravens 9y ago
    Sure, though we're talking about a 6 figure/mo cloud spend. That's dozens or hundreds of servers. (+network config etc) I run our tech stack, and I can stand up instances pretty easily, but I had to rack and stack, manage power, bring in leased lines, I'd be well outside of my skill set.
ExpensiveCloud 9y ago
Working at a large company migrating from our own datacenters to AWS where our monthly bill is approaching 8 figures. The higher ups were lured in with promises of easy elasticity that'd allow them to save money compared to our relatively overprovisioned DCs but the reality is that only some projects that benefit from this and certain teams which need large numbers of heavy duty servers for baseline usage are much more expensive. We have an internal team of 5-6 who are responsible just for building infrastructure on AWS for other teams and prodding teams to get into AWS.
bsenftner 9y ago
A few years ago, I built a compute-intensive REST API: applied machine learning that did 3D reconstruction of human heads from images. After careful spec'ing of the system's requirements, it was clear that any of the cloud services were exponentially more expensive than running my own hardware. I needed a $96K per month spend at AWS, the lowest of the cloud services at the time, or I could spend ~$50K and build a globally scaled server cluster of my own. People speak as if there is some issue with locating quality devOps individuals, but I seem to find them seeking work all the time. So I hired one to help me build the cluster: he re-visioned my specs, adding a hardware firewall, I purchased the hardware, we assembled the cluster at a local co-working space, and then we installed it into a local ISP, in a rented cabinet. Total expense to build was about $65K (less than one month of AWS), and the monthly cabinet rental was $600 a month. And the month to month maintenance was a whopping 5-10 minutes a few times a week. This is NOT HARD FOLKS and it SAVES A SHIT TON OF MONEY. Don't believe the cloud hype - it is all hype.
- dragonwriter 9y ago
  Most apps are much more ops intensive per unit of compute than what you describe, which makes paying cloud compute costs to offload some large fraction of the ops workload to someone else more worthwhile than it seems to have been for your app. And even in your case, I wonder if it would have gotten off the ground if you had to deal with infrastructure up front rather than as an efficiency move after the basic applocation was built, up, running, and stable: even if the cloud was mostly useful for ramping new systems up to the point where the system was basically stabilized and the infrastructure and ops needs were clear and could be assessed at leisure and efficiently provided with dedicated resources, that would still be a huge role.
rjeli 9y ago
One of the segment engineers addressed this here: https://news.ycombinator.com/item?id=13887218 https://news.ycombinator.com/item?id=13887218 tl;dr: They like the IaaS (ECS, Dynamo, ..), they need a lot of resources, and they're moving too fast to build it out in-house

djb_hackernews 9y ago

Depends on your workload. If you have a constant workload, then sure. But if you have variable workloads then AWS/cloud providers are a no brainer. Source: 6 figure/mo AWS spend. We use pretty much every infrastructure piece in AWS in multiple regions and to replicate that flexibility in our own data centers would be an incredible cost.

mattbillenstein 9y ago
Your workload doesn't need to be super constant either -- I think I ran the numbers some time ago, but for larger boxes, if you needed the machines for more than 8 hours a day, it was cheaper to run them in a DC and just leave them idle the rest of the time than to run them for that 8+ hours on AWS and shut them down for the rest of the time.

kevinburke 9y ago

At the very least you'd need people with experience running bare metal, delay product pipeline to work on the transition, train the team, build replacements for all of the AWS tools, rebuild this monitoring pipeline without Cloudwatch. Probably looking at $500k in engineering and staffing and training costs at the very least, just to get back to where you started, not to mention the progress you didn't make on the product.

bdcravens 9y ago

I would imagine that unless you have staff sitting on their hands, that level of hardware commitment would require at least one extra employee, generally at a cost of 6 figures a year.

Aeolun 9y ago
Not 6 figures a month though.
- bdcravens 9y ago
  Never said it was an equal cost. Merely pointing out that overhead expenses will cut into cost savings. Your cloud spend isn't just cost of hardware + cloud provider's profit margin.
bsenftner 9y ago
Nope, these days servers pretty much run themselves. One just needs to make sure hardware is not failing, and that is pretty much it. Really. Speaking from experience.
- bdcravens 9y ago
  Sure, though we're talking about a 6 figure/mo cloud spend. That's dozens or hundreds of servers. (+network config etc) I run our tech stack, and I can stand up instances pretty easily, but I had to rack and stack, manage power, bring in leased lines, I'd be well outside of my skill set.

ExpensiveCloud 9y ago

Working at a large company migrating from our own datacenters to AWS where our monthly bill is approaching 8 figures. The higher ups were lured in with promises of easy elasticity that'd allow them to save money compared to our relatively overprovisioned DCs but the reality is that only some projects that benefit from this and certain teams which need large numbers of heavy duty servers for baseline usage are much more expensive. We have an internal team of 5-6 who are responsible just for building infrastructure on AWS for other teams and prodding teams to get into AWS.

bsenftner 9y ago

A few years ago, I built a compute-intensive REST API: applied machine learning that did 3D reconstruction of human heads from images. After careful spec'ing of the system's requirements, it was clear that any of the cloud services were exponentially more expensive than running my own hardware. I needed a $96K per month spend at AWS, the lowest of the cloud services at the time, or I could spend ~$50K and build a globally scaled server cluster of my own. People speak as if there is some issue with locating quality devOps individuals, but I seem to find them seeking work all the time. So I hired one to help me build the cluster: he re-visioned my specs, adding a hardware firewall, I purchased the hardware, we assembled the cluster at a local co-working space, and then we installed it into a local ISP, in a rented cabinet. Total expense to build was about $65K (less than one month of AWS), and the monthly cabinet rental was $600 a month. And the month to month maintenance was a whopping 5-10 minutes a few times a week. This is NOT HARD FOLKS and it SAVES A SHIT TON OF MONEY. Don't believe the cloud hype - it is all hype.

dragonwriter 9y ago
Most apps are much more ops intensive per unit of compute than what you describe, which makes paying cloud compute costs to offload some large fraction of the ops workload to someone else more worthwhile than it seems to have been for your app. And even in your case, I wonder if it would have gotten off the ground if you had to deal with infrastructure up front rather than as an efficiency move after the basic applocation was built, up, running, and stable: even if the cloud was mostly useful for ramping new systems up to the point where the system was basically stabilized and the infrastructure and ops needs were clear and could be assessed at leisure and efficiently provided with dedicated resources, that would still be a huge role.

rjeli 9y ago

One of the segment engineers addressed this here: https://news.ycombinator.com/item?id=13887218 https://news.ycombinator.com/item?id=13887218 tl;dr: They like the IaaS (ECS, Dynamo, ..), they need a lot of resources, and they're moving too fast to build it out in-house