3 ms·

I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP). I can assure you they are radically different; autoscaling works differently

I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP). I can assure you they are radically different; autoscaling works differently, the load balancers work differently, the networking infrastructure is completely different, the failure modes and limits behaviors are different, the instances perform differently, observability is different, and they all suck in unique and different ways that we discover on a daily basis. Shit even AWS can't keep their regions consistent; each region has different products and features and they fail in different ways. If you are the one maintaining it it's a full time job handling all these edge cases, it's completely miserable and I wouldn't recommend it to anyone.

Infernal 3y ago
> I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP) Are you using AKS, EKS, GKE on those providers, or deploying your own k8s on top of the compute those providers offer? It sounds to me like the former.
- hosh 3y ago
  I’ve done smaller deployments on GKE and another on EKS, and I can tell you, they are different enough. It’s when you start having to autoscale, optimize resources by instance types, and manage network ingress that these quirks start really come out. The essential ideas are invariant across cloud providers though. But I enjoy working with Kubernetes.
  - Infernal 3y ago
    I should’ve been clearer about what I was getting at. I agree AKS, EKS, GKE etc (cloud gnostic k8s) are different enough to cause a growth of complexity when managing a mixed environment of them. The post I was replying to seemed to be saying (by analogy) “Linux is hard to manage because I run into all sorts of trouble trying to support a mixed environment of SuSE, Ubuntu and RHEL, therefore Linux is just too complicated”.
- gymbeaux 3y ago
  The latter wouldn’t make sense. In such a case, there’s already little value to being in the cloud, but to be in several?..
floomk 3y ago
Once you get to the scale of a million containers (or "apps") spread over multiple clouds then everything will be miserable.

Infernal 3y ago

> I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP) Are you using AKS, EKS, GKE on those providers, or deploying your own k8s on top of the compute those providers offer? It sounds to me like the former.

hosh 3y ago
I’ve done smaller deployments on GKE and another on EKS, and I can tell you, they are different enough. It’s when you start having to autoscale, optimize resources by instance types, and manage network ingress that these quirks start really come out. The essential ideas are invariant across cloud providers though. But I enjoy working with Kubernetes.
- Infernal 3y ago
  I should’ve been clearer about what I was getting at. I agree AKS, EKS, GKE etc (cloud gnostic k8s) are different enough to cause a growth of complexity when managing a mixed environment of them. The post I was replying to seemed to be saying (by analogy) “Linux is hard to manage because I run into all sorts of trouble trying to support a mixed environment of SuSE, Ubuntu and RHEL, therefore Linux is just too complicated”.
gymbeaux 3y ago
The latter wouldn’t make sense. In such a case, there’s already little value to being in the cloud, but to be in several?..

floomk 3y ago

Once you get to the scale of a million containers (or "apps") spread over multiple clouds then everything will be miserable.