> If you add Kubernetes to this setup you will need to make your machines double the size just the extra side-cars that you need make things functional add 3 ma

> If you add Kubernetes to this setup you will need to make your machines double the size just the extra side-cars that you need make things functional add 3 machines to be the control plane of Kubernetes, add 3 more machines for etcd cluster, add multiple internal services like ingress controllers and log and metrics collection. No, you really don't. I run multiple 3-node (i.e., the bare minimum to survive an AZ loss) clusters, and they are just fine. We run multiple workloads on them. It's three entire VMs of RAM and compute. If that's not sufficient, you need to dig into which of the services you're running on it are eating too much of whatever resource is falling short, and conclude through debugging whether that's a bug, or you need to scale up. That's the exact same procedure as a service on a VMs. If your counter-example is not doing "log and metrics collection" — that's an apples-to-oranges comparison. I run both k8s & raw VMs, and … both run these "side-cars". Not only do both run it, they're often the exact same process. Even the ingress: in most "service is deployed to VM" scenarios I've seen, often the service is fronted with nginx acting as a reverse proxy. That's not only fulfilling the same role as something like ingress-nginx, it's practically the same process. Most startups should probably go with a managed k8s, in which case they're likely not going to even have etcd in the cluster, nor more of the rest of the control plane. (IMO: don't choose AKS, but IMO the problem there is the A, not the K.) > You already are paying double of what you would have if you just excluded Kubernetes out of this because in a medium to small business there isn’t any feature of Kubernetes you are really using. You shouldn't be, because again, I think the k8s example is grossly over-provisioned. But I also think k8s's ability to move workloads between nodes easily lends it an advantage here: if I need to change the underlying makeup of the VMs, I can. That means that I can shrink-to-fit them. In my career eng spinning up VMs are far more likely to over-provision than to right-size or under-provision. > The normal setup would cost any CI/CD engineer a couple of days to setup and normally not more than an hour or 2 to debug infrastructure related issues. > Deploying on Kubernetes usually involves rendering templates, setting CI/CD runners, generating on the fly kubeconfigs for security and so much more that setting up CI/CD would take not less than a month to have something that is production grade ready. I've spent time on both. "Normal" (to a VM) CI/CD flows have pathological corner cases dependent on the state of the VM. Flux or Argo is not hard to set up, and the benefit of YAML is reproducibility: pods consistently start from a clean state. Much of the deploy work I've done with deployment scripts on VMs is re-inventing features that are part of k8s. YAML generation is straight-forward from the need to have it be configurable. One you understand the need for it … IDK what the problem people have here is. "setting CI/CD runners"? > The idea of deploying that many components together with different versions and different maintainers makes it impossible near to have an error free Kubernetes cluster, from network CNI to controllers to custom operators things tends to breakdown most of the https … this is always going to be a problem…? On a VM, I've got nginx, whatever handles certs, logs, metrics, security team's spyware^Wprotection etc. … all from different vendors, at different versions. "things tends to breakdown most of the https" ??? > It makes perfect sense with all the network encapsulation over the cloud network virtualization over the real network well things tend to get crazy. ??? > To be able to keep track of the versions of all components deployed on your cluster (same complaint) > And it should you are opting in for a setup intended for huge enterprises that have huge budgets and don’t mind having Kubernetes because it is actually benefiting them. This is begging the question. > Bigger attack surface While the API server is certainly a non-zero attack surface, this blows it out of proportion. People run salt on VMs, … see it's security track record: k8s fairs a far bit better. I've witnessed many orgs not set up some for of SSO, and try to manage VM access via SSH keys, and old keys do not get cleaned from nodes. The standard tools work for the API server: you can firewall it off, if it scares you? I would caution a young startup against running k8s: if you want to do it, you either need to have someone familiar with it who can teach, or eng who are capable of learning. If you have the all-too-common eng who don't like learning and who aren't familiar with it, you are going to have a bad time. But it isn't hard to learn: k8s has great documentation. But I've witnessed a lot of frustration from people who haven't RTFM.