3 ms·

Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at Stac

by mythz 3y ago

Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud. [1] https://stackexchange.com/performance https://stackexchange.com/performance

swozey 3y ago
I was a k8s contrib since 2015, version 1.1. I even worked at Rancher and Google Cloud. If you don't need absolutely granular control over a PAAS/SAAS (complex networking w/ circuit breaking yadda yadda, deep stack tracing, vms controlled by k8s (kubevirt etc), multi-tenancy in cpu or gpu) you don't need k8s and will absolutely flourish using a container solution like ECS. Use fargate and arm64 containers and you will save an absolute fortune. I dropped our AWS bill from $350k/mo to around $250k converting our largest apps to arm from x86. GKE is IMO the best k8s solution PAAS wise that exists, but quite frankly few companies need that much control and granularity in their infrastructure. My entire infrastructure now is AWS ECS and it autoscales and I literally never, ever, ever have had to troubleshoot it outside of my own configuration mishaps. I NEVER get on call alerts. I'm the Staff SRE at my corp.
- walth 3y ago
  Can completely agree. We went k8s several years ago and we battle the complexity continually. Want to change the pod spec to alter a variable under Vitess? That will be restarting several thousands of pods. DNS lookups failing as IPVS tuple collision? API server memory balooning to OOM due to watch/list cm storm? It all basically boils down to several FTE servicing K8s instead of building our platform. And on call is a nightmare.

by mythz 3y ago

swozey 3y ago

I was a k8s contrib since 2015, version 1.1. I even worked at Rancher and Google Cloud. If you don't need absolutely granular control over a PAAS/SAAS (complex networking w/ circuit breaking yadda yadda, deep stack tracing, vms controlled by k8s (kubevirt etc), multi-tenancy in cpu or gpu) you don't need k8s and will absolutely flourish using a container solution like ECS. Use fargate and arm64 containers and you will save an absolute fortune. I dropped our AWS bill from $350k/mo to around $250k converting our largest apps to arm from x86. GKE is IMO the best k8s solution PAAS wise that exists, but quite frankly few companies need that much control and granularity in their infrastructure. My entire infrastructure now is AWS ECS and it autoscales and I literally never, ever, ever have had to troubleshoot it outside of my own configuration mishaps. I NEVER get on call alerts. I'm the Staff SRE at my corp.

walth 3y ago
Can completely agree. We went k8s several years ago and we battle the complexity continually. Want to change the pod spec to alter a variable under Vitess? That will be restarting several thousands of pods. DNS lookups failing as IPVS tuple collision? API server memory balooning to OOM due to watch/list cm storm? It all basically boils down to several FTE servicing K8s instead of building our platform. And on call is a nightmare.