GKE is an abomination. Its reputation is completely undeserved. Sure there's some nice things out of the box, like authenticating to clusters with your Google/G

GKE is an abomination. Its reputation is completely undeserved. Sure there's some nice things out of the box, like authenticating to clusters with your Google/GCP account, but day 2 operations are a constant frustration. What sucks? 1. The Kubernetes Pod garbage collector is configured to be abominally slow, keeping terminated pods in the API server for far too long. This interferes with cluster monitoring by making it seem like there's a consistently high number of OOMKilled etc. pods rather than blipping as it happens. GCP support claims this is working as intended and recommends manually running a script to clean up the API server if it bothers you (this is a managed service?!). See e.g. https://stackoverflow.com/questions/75374590/why-kubernetes-keeps-pods-in-error-completed-status-preemptible-nodes-gke https://stackoverflow.com/questions/75374590/why-kubernetes-... . 2. The rest of the Kubernetes world moved on from kube-dns and on to CoreDNS. Not GKE! On GKE your two options are kube-dns and the GCP VPC-native Cloud DNS (i.e. Kubernetes service and pod records are listed in the private DNS zone for the VPC). Surprise surprise - if you pick Cloud DNS to help scale your cluster, because GCP isn't operating kube-dns well enough on its managed control plan, then you're on the hook for paying for the Cloud DNS zone as well, it's not included in the GKE cluster costs. See e.g. https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-dns https://cloud.google.com/kubernetes-engine/docs/how-to/cloud... . 3. GKE clusters automatically log to GCP Cloud Logging, the first 50 GB of which is free. Fair enough. But the ingestion price afterwards is a truly mind-boggling $0.50/GB! (https://cloud.google.com/logging/#section-7 https://cloud.google.com/logging/#section-7). How do you turn off GCP Cloud Logging so that you can ship your logs to a cheaper vendor instead? Nope, there's no first-class managed setting; all you get is a community tutorial (https://cloud.google.com/community/tutorials/kubernetes-engine-customize-fluentbit https://cloud.google.com/community/tutorials/kubernetes-engi...) that links to this GitHub configuration (https://github.com/GoogleCloudPlatform/community/blob/master/tutorials/kubernetes-engine-customize-fluentbit/kubernetes/fluentbit-configmap.yaml https://github.com/GoogleCloudPlatform/community/blob/master...) aaaaand good luck :) 4. No native IPv6. See e.g. https://stackoverflow.com/questions/64110542/has-anyone-identified-a-good-solution-for-ipv6-in-gke-google-game-servers-or-a https://stackoverflow.com/questions/64110542/has-anyone-iden... . AWS of course does support IPv6: https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.html https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.ht... And this is just off the top of my head.