We use it to expose not just a common, but mostly a simpler interface to data/engineering people via EKS/GKE to do things they commonly need to with less of a l

We use it to expose not just a common, but mostly a simpler interface to data/engineering people via EKS/GKE to do things they commonly need to with less of a learning curve, but we wouldn't run Kubernetes on bare metal, and we don't commonly move workloads between clouds (though we do run Kubernetes on both AWS and GCP). Turns out you need a lot of AWS/GCP knowledge even to run relatively simple apps, and even more for processing pipelines, and especially on AWS, every product has its own often steep learning curve, they don't necessarily share much. We find our k8s platforms offer a nice middle ground between siloing AWS/GCP knowledge and access away from engineers/data people and requiring everyone to fully understand the respective clouds, so they can have wide-ranging access and take care of things themselves. For example, it can be a huge pain to find out how to debug issues in native AWS/GCP products (Cloud Run, looking at you especially); with Kubernetes, common usage patterns share a lot more, e.g. debugging pretty much always involves pod logs, maybe exec'ing into a pod, reading and editing YAML in a familiar structure, be it a scheduled processing pipeline, app deployment, or whatever else, so we can train people how to do this once, and we find they're able to transfer that knowledge pretty well with a little pointing them to the right namespace/pod, and they can have all the access they need (depending on the cluster). Our own little tailored micro-clouds, if you will, but with a huge lot less effort than doing this from scratch, since a lot of the hard work involved is done by managed Kubernetes, and without much of the overhead of a cloud that has to work with 7823783 products involved. Similar things can be achieved by providing CDK templates, Terraform modules, well thought-out IAM policies, etc. etc., but this works well for us and appears to be a lot less hassle and complexity than those alternatives. It also helps security-wise that learning Kubernetes security once goes a very long way, and the concepts seem quite a bit more limited than for, say, running arbitrary things directly on AWS. Locking things up in separate clusters and namespaces can also add an additional layer to limit blast radii. Though we also use k8s because, especially on GCP, where we do the most data-intense work, strict GDPR compliance seems to be incredibly hard to achieve with much of their serverless offerings, it's incredibly easy to accidentally transfer data to the US without realizing, and there seems to be no mechanism to prevent that globally, though that better fits the category of things that cannot run on (serverless) Cloud. We still run many of these kind of workloads in the Cloud, but in GKE. Those that cannot run in cloud at all due to legal requirements don't run on Kubernetes, but directly on their own dedicated hardware on-prem. Inter-cloud portability is neat as well, but we wouldn't have picked k8s just for that.