4 ms·

I am consistently confused by all of the talk about how "hard" Kubernetes is. We spin up EKS. We install the newrelic and datadog log ingestion pods onto it, p

I am consistently confused by all of the talk about how "hard" Kubernetes is. We spin up EKS. We install the newrelic and datadog log ingestion pods onto it, provided in a nice "helm" format. We install a few other resources via helm, like external secrets, and external dns, and a few others. Kubernetes EKS runs like a champ. My company saves 100k/mo by dynamically scaling our cloud services, all of which are running on Kubernetes, to more efficiently use compute classes. My company has over 50 million unique users monthly. We have massive scale. Kubernetes just works for us and we only have 2 people maintaining it. What we gain is a unified platform with a consistent API for developing our services. And if we wanted to migrate elsewhere, it is one less thing to worry about. ¯\_(ツ)_/¯ Feels like some kind of hipster instinct to dislike the "cool new thing"... even though k8 has been around for years now and has been battle tested to the bone.

Turbots 4y ago
Your company "saves" over 100k/month paying WAY too much for EKS, which is extremely expensive. If you're at any decent scale (looks like you are), then switch to GKE, or switch to on-prem and buy some hardware + a Kubernetes distro like Mirantis/Openshift/Tanzu. Heck, go run k3s on Hetzner and you won't have that much more work, but save literally millions at the scale you're talking about.
- unity1001 4y ago
  > Heck, go run k3s on Hetzner The moment Hetzner starts providing managed DBs or managed kubernetes, they will wipe the floor with all the big 3...
- xyzzy_plugh 4y ago
  EKS is $0.10/hour, how is that "extremely expensive"? Or are you referring to an adjacent service?
drdaeman 4y ago
So, what do you do when one of your pods suddenly cannot connect to another, even though both nodes seem to be passing healthchecks and stuff? Spinning up a K8s cluster "in the cloud" is easy and everyone jumps on that happy "look how simple it all is" bandwagon, forgetting than it's just the beginning of a very long journey. There are millions of blog articles of varying quality that explain how easy it is, because it's very simple to spam search engines retelling a story how to click a couple buttons or do some basic Terraform/CloudFormation/whatever. And here's what they don't tell you - maintaining this machinery is still your business, because all you get is a bunch of provisioned node machines and a cookiecutter to spin it up. Plus a bunch of scripts to handle most common scenarios (scaling, upgrading, some basic troubleshooting, etc). The rest is either on your or tech support (if you pay extra for it). And if you have a sysadmin-for-hire contract anyway, then its them who should have an opinion what's easy and what's hard. Contracting other people is always relatively easy - compared to what they do.
rjh29 4y ago
Yeah it's easy for you because it's two people's full time job to maintain it? Many of us are having to learn it and use it in our spare time, or on top of our other work. We wouldn't necessarily know the best practices, or to use newrelic and datadog, or what to use for external secrets, external dns, how to diagnose and debug the issues which inevitably will occur when setting it up. Now this is true for doing it without k8s too, but somehow there was never a huge set of blog posts about "it's really hard to set up a load balancer and a secrets service and networking" but there is for k8s, so there must something intrinsic in either its design or its documentation that is causing that. I think it's probably that k8s is designed for google-scale deployments, so for most people the initial burst of complexity is a bit overwhelming.
sass_muffin 4y ago
Same, we use EKS and a very similar setup, our workload has some pretty high throughput and scaling requirements. Works amazing for our team, wouldn't change it for anything else at this point. Very low maintenance effort since AWS manages the K8s infra.

Turbots 4y ago

Your company "saves" over 100k/month paying WAY too much for EKS, which is extremely expensive. If you're at any decent scale (looks like you are), then switch to GKE, or switch to on-prem and buy some hardware + a Kubernetes distro like Mirantis/Openshift/Tanzu. Heck, go run k3s on Hetzner and you won't have that much more work, but save literally millions at the scale you're talking about.

unity1001 4y ago
> Heck, go run k3s on Hetzner The moment Hetzner starts providing managed DBs or managed kubernetes, they will wipe the floor with all the big 3...
xyzzy_plugh 4y ago
EKS is $0.10/hour, how is that "extremely expensive"? Or are you referring to an adjacent service?

drdaeman 4y ago

So, what do you do when one of your pods suddenly cannot connect to another, even though both nodes seem to be passing healthchecks and stuff? Spinning up a K8s cluster "in the cloud" is easy and everyone jumps on that happy "look how simple it all is" bandwagon, forgetting than it's just the beginning of a very long journey. There are millions of blog articles of varying quality that explain how easy it is, because it's very simple to spam search engines retelling a story how to click a couple buttons or do some basic Terraform/CloudFormation/whatever. And here's what they don't tell you - maintaining this machinery is still your business, because all you get is a bunch of provisioned node machines and a cookiecutter to spin it up. Plus a bunch of scripts to handle most common scenarios (scaling, upgrading, some basic troubleshooting, etc). The rest is either on your or tech support (if you pay extra for it). And if you have a sysadmin-for-hire contract anyway, then its them who should have an opinion what's easy and what's hard. Contracting other people is always relatively easy - compared to what they do.

rjh29 4y ago

Yeah it's easy for you because it's two people's full time job to maintain it? Many of us are having to learn it and use it in our spare time, or on top of our other work. We wouldn't necessarily know the best practices, or to use newrelic and datadog, or what to use for external secrets, external dns, how to diagnose and debug the issues which inevitably will occur when setting it up. Now this is true for doing it without k8s too, but somehow there was never a huge set of blog posts about "it's really hard to set up a load balancer and a secrets service and networking" but there is for k8s, so there must something intrinsic in either its design or its documentation that is causing that. I think it's probably that k8s is designed for google-scale deployments, so for most people the initial burst of complexity is a bit overwhelming.

sass_muffin 4y ago

Same, we use EKS and a very similar setup, our workload has some pretty high throughput and scaling requirements. Works amazing for our team, wouldn't change it for anything else at this point. Very low maintenance effort since AWS manages the K8s infra.