5 ms·

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And l

by cedws 3y ago

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.

xyzzy_plugh 3y ago
This isn't my experience at all. Maybe three or four years ago? Who exactly needs millions of lines of code?
- Spivak 3y ago
  I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc. And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
  - mschuster91 3y ago
    We have all of that neatly extracted into a Terraform module. Write it once and now EKS clusters are essentially disposable.
    - Solvency 3y ago
      You just added yet another Thing in that huge pile of things representing millions of lines of code. That's the point.
      - dvfjsdhgfv 3y ago
        Everything we run our workloads on is based on millions of LoCs, whether it's in the OS, in K8S, in is built-in or external kinds. If you decide to run K8S in AWS, you'll be better of using Karpenter, external-secrets and all these things as they will make your life easier in various ways.
      - lijok 3y ago
        Why is that inherently a problem? How many LOCs in the linux kernel again?
woleium 3y ago
kinda like openshift?
mardifoufs 3y ago
Millions upon millions of lines of code?! What? Can you specify what you were trying to do with it?
- cedws 3y ago
  Argo CD, Argo Rollouts, Vault, External Secrets, Cert Manager, Envoy, Velero, plus countless operators, plus a service mesh if you need it, the list goes on. If you're providing Kubernetes as a platform at any sort of scale you're going to need most of this stuff or some alternatives. This sums up to at least multiple million LOC. Then you have Kubernetes itself, containerd, etcd...
  - arccy 3y ago
    that's not much different from using the cloud PaaS offerings besides who runs that million lines and who gets the freedom/control for customization.
foofie 3y ago
> K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.
- WildGreenLeave 3y ago
  I run my own cluster too, it is managed by one terraform file which is maintained on GitHub [0]. Along with that I deploy everything on here with 1 shell script and a bunch of yaml manifests for my services. It's perfect for projects that are managed by one person (me). Everything is in git and reproducable. The only thing I am doing unconventional is that I didn't want to use github actions, so I use Kaniko to build my Docker containers inside my cluster. 0 https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
- cedws 3y ago
  If you're using a K8S cluster just to deploy a few web apps then it's not really a platform that you could provide to an engineering team within a medium-large company. You could probably run your stuff on ECS.
  - foofie 3y ago
    > If you're using a K8S cluster just to deploy a few web apps (...) It's really not about what I do and do not do with Kubernetes. It's on you to justify your "millions upon millions lines of code" claim because it is so outlandish and detached from reality that it says more about your work than about Kubernetes. I repeat: I only need a few dozen lines of kustomize scripts to release whole web apps. Simple code. Easy peasy. What mess are you doing to require "millions upon millions" lines of code?
    - cedws 3y ago
      You are missing the point. I recommend you look into Platform Engineering and what it involves.
mise_en_place 3y ago
kubeadm + fabric + helm got me 99% of the way there. My direct report, a junior engineer, wrote the entire helm chart from our docker-compose. It will not entirely replace our remote environment but it is nice to have something in between our SDK and remote deployed infra. Not sure what you meant by security; could you elaborate? I just needed to expose one port to the public internet.

by cedws 3y ago

xyzzy_plugh 3y ago

This isn't my experience at all. Maybe three or four years ago? Who exactly needs millions of lines of code?

Spivak 3y ago
I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc. And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
- mschuster91 3y ago
  We have all of that neatly extracted into a Terraform module. Write it once and now EKS clusters are essentially disposable.
  - Solvency 3y ago
    You just added yet another Thing in that huge pile of things representing millions of lines of code. That's the point.
    - dvfjsdhgfv 3y ago
      Everything we run our workloads on is based on millions of LoCs, whether it's in the OS, in K8S, in is built-in or external kinds. If you decide to run K8S in AWS, you'll be better of using Karpenter, external-secrets and all these things as they will make your life easier in various ways.
    - lijok 3y ago
      Why is that inherently a problem? How many LOCs in the linux kernel again?

woleium 3y ago

kinda like openshift?

mardifoufs 3y ago

Millions upon millions of lines of code?! What? Can you specify what you were trying to do with it?

cedws 3y ago
Argo CD, Argo Rollouts, Vault, External Secrets, Cert Manager, Envoy, Velero, plus countless operators, plus a service mesh if you need it, the list goes on. If you're providing Kubernetes as a platform at any sort of scale you're going to need most of this stuff or some alternatives. This sums up to at least multiple million LOC. Then you have Kubernetes itself, containerd, etcd...
- arccy 3y ago
  that's not much different from using the cloud PaaS offerings besides who runs that million lines and who gets the freedom/control for customization.

foofie 3y ago

> K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.

WildGreenLeave 3y ago
I run my own cluster too, it is managed by one terraform file which is maintained on GitHub [0]. Along with that I deploy everything on here with 1 shell script and a bunch of yaml manifests for my services. It's perfect for projects that are managed by one person (me). Everything is in git and reproducable. The only thing I am doing unconventional is that I didn't want to use github actions, so I use Kaniko to build my Docker containers inside my cluster. 0 https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
cedws 3y ago
If you're using a K8S cluster just to deploy a few web apps then it's not really a platform that you could provide to an engineering team within a medium-large company. You could probably run your stuff on ECS.
- foofie 3y ago
  > If you're using a K8S cluster just to deploy a few web apps (...) It's really not about what I do and do not do with Kubernetes. It's on you to justify your "millions upon millions lines of code" claim because it is so outlandish and detached from reality that it says more about your work than about Kubernetes. I repeat: I only need a few dozen lines of kustomize scripts to release whole web apps. Simple code. Easy peasy. What mess are you doing to require "millions upon millions" lines of code?
  - cedws 3y ago
    You are missing the point. I recommend you look into Platform Engineering and what it involves.

mise_en_place 3y ago

kubeadm + fabric + helm got me 99% of the way there. My direct report, a junior engineer, wrote the entire helm chart from our docker-compose. It will not entirely replace our remote environment but it is nice to have something in between our SDK and remote deployed infra. Not sure what you meant by security; could you elaborate? I just needed to expose one port to the public internet.