4 ms·

It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers

It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers is a good idea imho. What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do. Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.

pphysch 4y ago
A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis. Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required. Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
- indymike 4y ago
  I've had great luck just leasing what I need from a company who already has the DC and is willing to manage the hardware as part of the lease.
- kkielhofner 4y ago
  Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW. For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute. For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate). I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them. That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
  - tatersolid 4y ago
    Remote hands, maintenance contracts, and employees who will do network/firmware/firewall config, patching and maintenance aren’t included. Also there’s no software costs… if you’re going to compare with the cloud you need to include VMware/Microsoft suite licenses, maintenance, and operators to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.
    - kkielhofner 4y ago
      Remote hands from co-lo facilities are typically free for simple activities like hitting a switch, plugging something in, etc. Hourly rates are extremely reasonable and initial racking is often included. Reality is if you’re using remote hands after the initial install you’re doing it wrong - out of band management, PXE boot to full machine bootstrap via ansible, K8, etc isn’t any more complicated than managing a bunch of VPCs, cloud firewalls, routing, cloud templates/terraform, etc. Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three/five years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light. AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective. As one example for the most "unreliable" component the BackBlaze hard drive stats tell the story - they have a roughly 1% failure rate across > 100k consumer level mechanical drives[0] in service for a long as 92 months. These "failure prone" consumer drives aren't even available from server vendors. This is why their pricing is at least 1/10th that of big cloud. VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely. I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash. For the rest of the dev/devops teams shipping a container is shipping a container. The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left over for the rounding error miscellaneous costs often noted. Cloud egress bandwidth pricing markups alone can pay for it when you’re pushing the kind of traffic that would warrant this approach. I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”). For most of the "cloud only" generation their only experience with hardware is that time their laptop died and needless to say server grade hardware in temperature controlled facilities on conditioned power is infinitely more reliable than a consumer laptop getting thrown around. Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud costs. Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cottage industry of consultants and startups that exist solely to help orgs try to understand their cloud spend and attempt to optimize cost. Hardware leasing, fixed price co-lo, power, and bandwidth results in cost controls that just aren’t possible with cloud. Almost every startup has at least one story of scrambling to figure out what happened when they blow through their monthly cloud budget in many cases literally overnight - or begging their cloud provider to write off or steeply discount a monthly invoice that was wildly beyond predicted cost (and in many cases they literally wouldn't be able to pay). In multiple rounds of investment, etc in startups I’ve often had the experience during technical and financial due-diligence where they look at hosting costs and literally can’t believe how cost effective and performant it is vs the “all cloud always” companies in their portfolio. At a level of scale to even consider it hardware and co-lo is practically cheating. [0] - https://www.backblaze.com/blog/backblaze-drive-stats-for-2022/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202...
hdjjhhvvhga 4y ago
> What I am missing though in the monthly cost is the personnel cost. It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.
- deleted 4y ago
  [deleted]
- kevincox 4y ago
  I'm pretty confident that at small scale AWS requires less operating costs that colo. Things as simple as s3 take lots of work to setup and run in a colo (well probably at least 3 locations if you want really good durability) and things like backups quickly build on that. Even a single computer will have occasional hardware failures, system installs and more. You usually can't restore an entire disk from backup like you can on AWS if a DB upgrade goes wrong. But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service. It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are. As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.
drunkenmagician 4y ago
Nope, senior devops / sys admin folk here in Singapore are more in the 7K - 8.5K (usd) range. Unless of course the employer is paying FFANG level salaries

pphysch 4y ago

A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis. Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required. Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.

indymike 4y ago
I've had great luck just leasing what I need from a company who already has the DC and is willing to manage the hardware as part of the lease.
kkielhofner 4y ago
Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW. For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute. For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate). I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them. That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
- tatersolid 4y ago
  Remote hands, maintenance contracts, and employees who will do network/firmware/firewall config, patching and maintenance aren’t included. Also there’s no software costs… if you’re going to compare with the cloud you need to include VMware/Microsoft suite licenses, maintenance, and operators to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.
  - kkielhofner 4y ago
    Remote hands from co-lo facilities are typically free for simple activities like hitting a switch, plugging something in, etc. Hourly rates are extremely reasonable and initial racking is often included. Reality is if you’re using remote hands after the initial install you’re doing it wrong - out of band management, PXE boot to full machine bootstrap via ansible, K8, etc isn’t any more complicated than managing a bunch of VPCs, cloud firewalls, routing, cloud templates/terraform, etc. Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three/five years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light. AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective. As one example for the most "unreliable" component the BackBlaze hard drive stats tell the story - they have a roughly 1% failure rate across > 100k consumer level mechanical drives[0] in service for a long as 92 months. These "failure prone" consumer drives aren't even available from server vendors. This is why their pricing is at least 1/10th that of big cloud. VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely. I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash. For the rest of the dev/devops teams shipping a container is shipping a container. The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left over for the rounding error miscellaneous costs often noted. Cloud egress bandwidth pricing markups alone can pay for it when you’re pushing the kind of traffic that would warrant this approach. I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”). For most of the "cloud only" generation their only experience with hardware is that time their laptop died and needless to say server grade hardware in temperature controlled facilities on conditioned power is infinitely more reliable than a consumer laptop getting thrown around. Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud costs. Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cottage industry of consultants and startups that exist solely to help orgs try to understand their cloud spend and attempt to optimize cost. Hardware leasing, fixed price co-lo, power, and bandwidth results in cost controls that just aren’t possible with cloud. Almost every startup has at least one story of scrambling to figure out what happened when they blow through their monthly cloud budget in many cases literally overnight - or begging their cloud provider to write off or steeply discount a monthly invoice that was wildly beyond predicted cost (and in many cases they literally wouldn't be able to pay). In multiple rounds of investment, etc in startups I’ve often had the experience during technical and financial due-diligence where they look at hosting costs and literally can’t believe how cost effective and performant it is vs the “all cloud always” companies in their portfolio. At a level of scale to even consider it hardware and co-lo is practically cheating. [0] - https://www.backblaze.com/blog/backblaze-drive-stats-for-2022/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202...

hdjjhhvvhga 4y ago

> What I am missing though in the monthly cost is the personnel cost. It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.

deleted 4y ago
[deleted]
kevincox 4y ago
I'm pretty confident that at small scale AWS requires less operating costs that colo. Things as simple as s3 take lots of work to setup and run in a colo (well probably at least 3 locations if you want really good durability) and things like backups quickly build on that. Even a single computer will have occasional hardware failures, system installs and more. You usually can't restore an entire disk from backup like you can on AWS if a DB upgrade goes wrong. But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service. It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are. As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.

drunkenmagician 4y ago

Nope, senior devops / sys admin folk here in Singapore are more in the 7K - 8.5K (usd) range. Unless of course the employer is paying FFANG level salaries