On-prem k8s is almost always a bad idea. Things like storage or traffic routing become uncertain/hacky at best or at worst just straight up don’t work. I’ve l

On-prem k8s is almost always a bad idea. Things like storage or traffic routing become uncertain/hacky at best or at worst just straight up don’t work. I’ve literally seen many on-prem k8s deployments where there was no support for persistent volume claims because of storage - whether the enterprise storage solution wasn’t supported by k8s or because there was no way to guarantee storage (HDDs being a capital expense and therefore purchased ahead of time). Examples of friction off the top of my head: * storage - who knows if there’s actually enough space to provision and attach a volume * exposing a service externally * load balancers managed by a different team (such as networking) and their process or tech environment not setup to support k8s * certificates need to be requested and signed by approved enterprise certificate authority * network security restrictions - lengthy process for firewall change requests or similar to expose a service externally (also related to point above on certificates) I’ve also seen silly deployments where k8s runs inside VMware virtual machines - why not just run k8s directly on hardware if on-prem? Virtual machines just add a ton of unnecessary and expensive overhead. Personally, my preferred way of doing k8s: 1. GKE (Google Kubernetes Engine) - the easiest and cheapest managed option 2. kops Hot take here but I personally think EKS is convoluted and complicated as hell. I’ve not used AKS but in the past I’ve seen limitations such as network configuration that became problematic - couldn’t route traffic properly because IP space overlapped with corporate IP ranges already used by VPN and/or existing vnets, though it’s been a couple years since I first saw this specific example.