4 ms·

> We could go down to half that price or less if we rolled our own Kubernetes cluster, but the added labor might negate the savings. It's so refreshing to see

> We could go down to half that price or less if we rolled our own Kubernetes cluster, but the added labor might negate the savings. It's so refreshing to see someone taking this sort of complexity into account when planning new infrastructure. There's a HUGE cost to training and supporting a new infra paradigm that I often see decision makers ignore completely... in their defense it's a difficult thing to predict but that's not a reason to ignore the overhead!

api 6y ago
Always try to do a spreadsheet and estimate the actual cost. The last time we did the math it was kind of a toss-up between roll your own bare metal cluster and GKS when labor costs were considered, but those are estimated labor costs and we also didn't want to deal with the hassle. GKS meanwhile destroys Amazon, so all the big clouds are not equal. We decided we might make the leap when the savings for bare metal over GKE are 2X or more, since that leaves a good margin for under-estimating labor (which is probably the case). I edited the parent to add a point about rolling your own making progressively more sense the bigger you get. That's because modern devops lets you leverage a skilled admin across a big deployment, so the labor cost becomes somewhat fixed and allows you to leverage the economy of scale and the cost advantages of bare metal (to a point). But do your own estimates! There are many details. One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I italicized those because damn do the big cloud providers ever rape you on bandwidth. The same is true to a less absurd degree for raw CPU power. It mystifies me when I see people doing deep learning on AWS. Storage on the other hand is far cheaper on Amazon and other major cloud providers, especially when you factor in the cost of achieving high reliability and uptime. If you are warehousing a ton of data, you might never want to leave these (unless you actually get bit enough to build your own racks!). Edit: caveat on storage: check out Backblaze B2 and using it from a nearby bare metal data center if you are mostly warehousing and not heavily accessing. Again: there are many many devils in these details, so do your own shopping and cost estimates for your unique workload and factor in everything: compute, storage, bandwidth, labor, uptime requirements, agility, rapid auto-scaling needs, geographic location and/or multi-location needs, etc. Hosting/cloud is a huge market with a dizzying array of choices each making sense for different mixtures of customer profiles. Expect to spend a day or two just surfing around and shopping. FYI the bare metal hosts we've had good luck with are packet.net, datapacket.com, and OVH, in approximately that order. (For OVH we have only used their bare metal hosts and can't comment on their other services.) There are others. We had lots of odd issues with Hetzner but they are dirt cheap partly because they cut corners like using desktop-grade hardware. Hetzner may actually make sense if you need a ton of super-cheap compute and have a load that is very tolerant of node failures, since as far as I can see nobody sells CPU as cheap as they do. If I were doing a ton of deep learning model training I'd consider using Hetzner for raw compute power cheap and I'd treat the nodes operationally like on-demand/spot instances. You might look at Digital Ocean and Vultr too. DO is adding some managed services and apparently they're decent. In the past the simple "we give you a VM" hosts have occupied an uncanny valley though: not as cheap in price/performance terms as bare metal "rent-a-box" hosting, and not rich enough in services and value-adds to compete with big cloud.
- folkhack 6y ago
  > One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I work in the ISP space specifically with folks who help place businesses in datacenters. This is 100% accurate - sometimes it can literally be pennies on the dollar for egress. > Storage on the other hand is far cheaper Agree - S3 will always be a backbone component of my toolbox when building out infrastructure. It's been solid from day-1 when I was in my early 20's using it on remix competition sites (wavs are big... 10-20 wavs per-song oh god). The awe still hasn't worn off for me as to how quick I can scale large-scale web-friendly/secure storage with tools like that. --- On the VPS hosts... I'm weirdly loyal to them and have kept infra on Linode for years. I also have had extraordinary positive experiences with DO as well. I find they have the customer service that solutions like AWS lack for the little guys... if we're talking about a big spend then yea I will totally go to a more formal provider, but for my stuff I prefer knowing that there's solid engineers on the hosting side that I can actually talk to. I'm REALLY excited to see folks like Linode/DO support things like S3-compatible APIs, K8s, etc. I think that's important for those solutions to stay relevant. --- Thanks for such a great response - conversations like these are invaluable, and I've definitely learned from this! I am VERY interested in Vultr - totally could have used them 5-6 months ago for GIS crunching (fiber maps, lit building lists, geocoding, census blocks, etc). Once I open that up as a service to my users I'll likely be on the hunt for SSD-backed high-compute on-demand instances.
  - kyuudou 6y ago
    DO also has great overall documentation quality and standards, IMO.

api 6y ago

Always try to do a spreadsheet and estimate the actual cost. The last time we did the math it was kind of a toss-up between roll your own bare metal cluster and GKS when labor costs were considered, but those are estimated labor costs and we also didn't want to deal with the hassle. GKS meanwhile destroys Amazon, so all the big clouds are not equal. We decided we might make the leap when the savings for bare metal over GKE are 2X or more, since that leaves a good margin for under-estimating labor (which is probably the case). I edited the parent to add a point about rolling your own making progressively more sense the bigger you get. That's because modern devops lets you leverage a skilled admin across a big deployment, so the labor cost becomes somewhat fixed and allows you to leverage the economy of scale and the cost advantages of bare metal (to a point). But do your own estimates! There are many details. One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I italicized those because damn do the big cloud providers ever rape you on bandwidth. The same is true to a less absurd degree for raw CPU power. It mystifies me when I see people doing deep learning on AWS. Storage on the other hand is far cheaper on Amazon and other major cloud providers, especially when you factor in the cost of achieving high reliability and uptime. If you are warehousing a ton of data, you might never want to leave these (unless you actually get bit enough to build your own racks!). Edit: caveat on storage: check out Backblaze B2 and using it from a nearby bare metal data center if you are mostly warehousing and not heavily accessing. Again: there are many many devils in these details, so do your own shopping and cost estimates for your unique workload and factor in everything: compute, storage, bandwidth, labor, uptime requirements, agility, rapid auto-scaling needs, geographic location and/or multi-location needs, etc. Hosting/cloud is a huge market with a dizzying array of choices each making sense for different mixtures of customer profiles. Expect to spend a day or two just surfing around and shopping. FYI the bare metal hosts we've had good luck with are packet.net, datapacket.com, and OVH, in approximately that order. (For OVH we have only used their bare metal hosts and can't comment on their other services.) There are others. We had lots of odd issues with Hetzner but they are dirt cheap partly because they cut corners like using desktop-grade hardware. Hetzner may actually make sense if you need a ton of super-cheap compute and have a load that is very tolerant of node failures, since as far as I can see nobody sells CPU as cheap as they do. If I were doing a ton of deep learning model training I'd consider using Hetzner for raw compute power cheap and I'd treat the nodes operationally like on-demand/spot instances. You might look at Digital Ocean and Vultr too. DO is adding some managed services and apparently they're decent. In the past the simple "we give you a VM" hosts have occupied an uncanny valley though: not as cheap in price/performance terms as bare metal "rent-a-box" hosting, and not rich enough in services and value-adds to compete with big cloud.

folkhack 6y ago
> One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I work in the ISP space specifically with folks who help place businesses in datacenters. This is 100% accurate - sometimes it can literally be pennies on the dollar for egress. > Storage on the other hand is far cheaper Agree - S3 will always be a backbone component of my toolbox when building out infrastructure. It's been solid from day-1 when I was in my early 20's using it on remix competition sites (wavs are big... 10-20 wavs per-song oh god). The awe still hasn't worn off for me as to how quick I can scale large-scale web-friendly/secure storage with tools like that. --- On the VPS hosts... I'm weirdly loyal to them and have kept infra on Linode for years. I also have had extraordinary positive experiences with DO as well. I find they have the customer service that solutions like AWS lack for the little guys... if we're talking about a big spend then yea I will totally go to a more formal provider, but for my stuff I prefer knowing that there's solid engineers on the hosting side that I can actually talk to. I'm REALLY excited to see folks like Linode/DO support things like S3-compatible APIs, K8s, etc. I think that's important for those solutions to stay relevant. --- Thanks for such a great response - conversations like these are invaluable, and I've definitely learned from this! I am VERY interested in Vultr - totally could have used them 5-6 months ago for GIS crunching (fiber maps, lit building lists, geocoding, census blocks, etc). Once I open that up as a service to my users I'll likely be on the hunt for SSD-backed high-compute on-demand instances.
- kyuudou 6y ago
  DO also has great overall documentation quality and standards, IMO.