14 ms·

> Amazon has found a way to attach a profit model to sophomore developers' tendency to over-engineer absolutely everything. Amen. It's also becoming the "safe"

> Amazon has found a way to attach a profit model to sophomore developers' tendency to over-engineer absolutely everything. Amen. It's also becoming the "safe" thing to do to put all of your eggs in the Amazon basket. Also just think about all of that data we're giving them... gah. They're laughing all the way to the bank.

api 6y ago
Meanwhile for companies like ours not doing this is a competitive advantage. Peoples' jaws drop when we tell them how insanely little we spend on cloud for hundreds of thousands of concurrent users and a lot of stuff going on. The secret: a la carte bare metal (packet.net, datapacket.com) for high-load and especially high-bandwidth stuff combined with a simple not-over-engineered services deployment in Google GKS with instances tuned for our work load (high CPU, low memory) so we are not paying for RAM we do not need. We are really only using Kubernetes to keep stuff up and auto-scale. We are not even close to using all K8S's features. GKS is most of the price tag. We could go down to about 1/4 the monthly cost if we rolled our own Kubernetes cluster with Rancher or something at packet.net, but the added labor might negate the savings. We may still do this if we grow significantly, since at larger scales the added labor may be justified by geometric cost savings. In general the bigger you are the more sense it makes to roll your own. Small and growing: use managed stuff. Bigger: consider bare metal and DIY clusters. Even bigger: co-location can start to make sense. HUGE: build your own data center! It's always a pure spreadsheet decision though. Also keep in mind that you can sometimes split your stack and do managed for one part and bare metal for another. For us the split is between things that are micro-service based, database backed, and need to auto-scale vs. dumb simple bandwidth and CPU pumping services that are trivial to deploy and require little attention. The latter is what goes on bare metal for massive operational cost savings. The former goes in GKS for labor cost savings. ... and yes, it's very stable, and probably more stable than finicky over-engineered Rube Goldberg machine AWS deployments that engineers constantly have to fiddle with.
- folkhack 6y ago
  > We could go down to half that price or less if we rolled our own Kubernetes cluster, but the added labor might negate the savings. It's so refreshing to see someone taking this sort of complexity into account when planning new infrastructure. There's a HUGE cost to training and supporting a new infra paradigm that I often see decision makers ignore completely... in their defense it's a difficult thing to predict but that's not a reason to ignore the overhead!
  - api 6y ago
    Always try to do a spreadsheet and estimate the actual cost. The last time we did the math it was kind of a toss-up between roll your own bare metal cluster and GKS when labor costs were considered, but those are estimated labor costs and we also didn't want to deal with the hassle. GKS meanwhile destroys Amazon, so all the big clouds are not equal. We decided we might make the leap when the savings for bare metal over GKE are 2X or more, since that leaves a good margin for under-estimating labor (which is probably the case). I edited the parent to add a point about rolling your own making progressively more sense the bigger you get. That's because modern devops lets you leverage a skilled admin across a big deployment, so the labor cost becomes somewhat fixed and allows you to leverage the economy of scale and the cost advantages of bare metal (to a point). But do your own estimates! There are many details. One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I italicized those because damn do the big cloud providers ever rape you on bandwidth. The same is true to a less absurd degree for raw CPU power. It mystifies me when I see people doing deep learning on AWS. Storage on the other hand is far cheaper on Amazon and other major cloud providers, especially when you factor in the cost of achieving high reliability and uptime. If you are warehousing a ton of data, you might never want to leave these (unless you actually get bit enough to build your own racks!). Edit: caveat on storage: check out Backblaze B2 and using it from a nearby bare metal data center if you are mostly warehousing and not heavily accessing. Again: there are many many devils in these details, so do your own shopping and cost estimates for your unique workload and factor in everything: compute, storage, bandwidth, labor, uptime requirements, agility, rapid auto-scaling needs, geographic location and/or multi-location needs, etc. Hosting/cloud is a huge market with a dizzying array of choices each making sense for different mixtures of customer profiles. Expect to spend a day or two just surfing around and shopping. FYI the bare metal hosts we've had good luck with are packet.net, datapacket.com, and OVH, in approximately that order. (For OVH we have only used their bare metal hosts and can't comment on their other services.) There are others. We had lots of odd issues with Hetzner but they are dirt cheap partly because they cut corners like using desktop-grade hardware. Hetzner may actually make sense if you need a ton of super-cheap compute and have a load that is very tolerant of node failures, since as far as I can see nobody sells CPU as cheap as they do. If I were doing a ton of deep learning model training I'd consider using Hetzner for raw compute power cheap and I'd treat the nodes operationally like on-demand/spot instances. You might look at Digital Ocean and Vultr too. DO is adding some managed services and apparently they're decent. In the past the simple "we give you a VM" hosts have occupied an uncanny valley though: not as cheap in price/performance terms as bare metal "rent-a-box" hosting, and not rich enough in services and value-adds to compete with big cloud.

api 6y ago

Meanwhile for companies like ours not doing this is a competitive advantage. Peoples' jaws drop when we tell them how insanely little we spend on cloud for hundreds of thousands of concurrent users and a lot of stuff going on. The secret: a la carte bare metal (packet.net, datapacket.com) for high-load and especially high-bandwidth stuff combined with a simple not-over-engineered services deployment in Google GKS with instances tuned for our work load (high CPU, low memory) so we are not paying for RAM we do not need. We are really only using Kubernetes to keep stuff up and auto-scale. We are not even close to using all K8S's features. GKS is most of the price tag. We could go down to about 1/4 the monthly cost if we rolled our own Kubernetes cluster with Rancher or something at packet.net, but the added labor might negate the savings. We may still do this if we grow significantly, since at larger scales the added labor may be justified by geometric cost savings. In general the bigger you are the more sense it makes to roll your own. Small and growing: use managed stuff. Bigger: consider bare metal and DIY clusters. Even bigger: co-location can start to make sense. HUGE: build your own data center! It's always a pure spreadsheet decision though. Also keep in mind that you can sometimes split your stack and do managed for one part and bare metal for another. For us the split is between things that are micro-service based, database backed, and need to auto-scale vs. dumb simple bandwidth and CPU pumping services that are trivial to deploy and require little attention. The latter is what goes on bare metal for massive operational cost savings. The former goes in GKS for labor cost savings. ... and yes, it's very stable, and probably more stable than finicky over-engineered Rube Goldberg machine AWS deployments that engineers constantly have to fiddle with.

folkhack 6y ago
> We could go down to half that price or less if we rolled our own Kubernetes cluster, but the added labor might negate the savings. It's so refreshing to see someone taking this sort of complexity into account when planning new infrastructure. There's a HUGE cost to training and supporting a new infra paradigm that I often see decision makers ignore completely... in their defense it's a difficult thing to predict but that's not a reason to ignore the overhead!
- api 6y ago
  Always try to do a spreadsheet and estimate the actual cost. The last time we did the math it was kind of a toss-up between roll your own bare metal cluster and GKS when labor costs were considered, but those are estimated labor costs and we also didn't want to deal with the hassle. GKS meanwhile destroys Amazon, so all the big clouds are not equal. We decided we might make the leap when the savings for bare metal over GKE are 2X or more, since that leaves a good margin for under-estimating labor (which is probably the case). I edited the parent to add a point about rolling your own making progressively more sense the bigger you get. That's because modern devops lets you leverage a skilled admin across a big deployment, so the labor cost becomes somewhat fixed and allows you to leverage the economy of scale and the cost advantages of bare metal (to a point). But do your own estimates! There are many details. One detail is that bare metal bandwidth (especially egress) is radically, ridiculously, nonsensically cheaper than any of the big cloud providers. I italicized those because damn do the big cloud providers ever rape you on bandwidth. The same is true to a less absurd degree for raw CPU power. It mystifies me when I see people doing deep learning on AWS. Storage on the other hand is far cheaper on Amazon and other major cloud providers, especially when you factor in the cost of achieving high reliability and uptime. If you are warehousing a ton of data, you might never want to leave these (unless you actually get bit enough to build your own racks!). Edit: caveat on storage: check out Backblaze B2 and using it from a nearby bare metal data center if you are mostly warehousing and not heavily accessing. Again: there are many many devils in these details, so do your own shopping and cost estimates for your unique workload and factor in everything: compute, storage, bandwidth, labor, uptime requirements, agility, rapid auto-scaling needs, geographic location and/or multi-location needs, etc. Hosting/cloud is a huge market with a dizzying array of choices each making sense for different mixtures of customer profiles. Expect to spend a day or two just surfing around and shopping. FYI the bare metal hosts we've had good luck with are packet.net, datapacket.com, and OVH, in approximately that order. (For OVH we have only used their bare metal hosts and can't comment on their other services.) There are others. We had lots of odd issues with Hetzner but they are dirt cheap partly because they cut corners like using desktop-grade hardware. Hetzner may actually make sense if you need a ton of super-cheap compute and have a load that is very tolerant of node failures, since as far as I can see nobody sells CPU as cheap as they do. If I were doing a ton of deep learning model training I'd consider using Hetzner for raw compute power cheap and I'd treat the nodes operationally like on-demand/spot instances. You might look at Digital Ocean and Vultr too. DO is adding some managed services and apparently they're decent. In the past the simple "we give you a VM" hosts have occupied an uncanny valley though: not as cheap in price/performance terms as bare metal "rent-a-box" hosting, and not rich enough in services and value-adds to compete with big cloud.