3 ms·

> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is

> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing. for example: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg... no more than 5k nodes. Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size. Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs. I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase. for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.

russellendicott 4y ago
Yeah, I don't know if it's because ECS was my first container orchestration experience but every time I look at teams trying to do k8s on AWS I think how much easier ECS would be.
- 015a 4y ago
  The complexity difference between bog-standard ECS+Fargate and EKS+Fargate deployments rounds down pretty small. Biggest I've seen: ALB integration, IAM integration, and maybe certificate management. Most of that stuff is out-of-box on ECS, but on EKS you need some extra containers or configuration to watch the K8s API and provision stuff for you (if you want to use it; you can also just go pure-k8s) (edit: just to be clear, they provide all this for you; its not out-of-box, but its easy-to-add-box e.g. [1]). An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it. An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless. Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar. [1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...
  - russellendicott 4y ago
    Agreed. I had the displeasure of doing CodeDeploy to ECS Fargate a few weeks ago for a side project and IMO it was overly complex.
    - 015a 4y ago
      Yup. You can get pretty dang far with just "the task definition is in cloudformation, so update the image in cloudformation and submit the template". For a bit more complexity, add ALB routing weights in, and the case for CodeDeploy is kind of weak. My statement that ECS needs it was probably overzealous; for most situations it doesn't, and if there's some crazy functionality it does I'm not aware of, you'd probably need something else in k8s as well (e.g. istio).
015a 4y ago
Right; and I don't feel that's a knock against K8s; its just the trade-offs it decided to make. A true Google-scale solution would be far worse to use, you can be sure of that. K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there). All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward. Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.
- kuschku 4y ago
  I'm using k8s even for my personal 2 node cluster. It's just so convenient to be able to use all the automation tools. I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc. At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.
arunc 4y ago
Curious how you ran 36k node cluster. Did it involve elastic scaling, etc? What are the alternatives for k8s?
- KaiserPro 4y ago
  Home grown: https://www.semanticscholar.org/paper/Robust-large-scale-rendering%3A-the-FQ-renderfarm-Hills-Vanns/6102a9bf5f3e54a98c87bb291bc22b96f20c2c83 https://www.semanticscholar.org/paper/Robust-large-scale-ren... off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alfred_Getting_Started.html https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf... although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high https://renderman.pixar.com/tractor https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/ https://www.opencue.io/ but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s. These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough. Nowadays, I'd use AWS batch, or at a push airflow.
singron 4y ago
Google's Borg clusters don't have wildly larger sizes than that despite having more years of development and a lot of motivation at that scale. They instead have a lot of clusters and transparent inter-cluster network topologies (i.e. you can choose clusters with very high bandwidths to each other). The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.

russellendicott 4y ago

Yeah, I don't know if it's because ECS was my first container orchestration experience but every time I look at teams trying to do k8s on AWS I think how much easier ECS would be.

015a 4y ago
The complexity difference between bog-standard ECS+Fargate and EKS+Fargate deployments rounds down pretty small. Biggest I've seen: ALB integration, IAM integration, and maybe certificate management. Most of that stuff is out-of-box on ECS, but on EKS you need some extra containers or configuration to watch the K8s API and provision stuff for you (if you want to use it; you can also just go pure-k8s) (edit: just to be clear, they provide all this for you; its not out-of-box, but its easy-to-add-box e.g. [1]). An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it. An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless. Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar. [1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...
- russellendicott 4y ago
  Agreed. I had the displeasure of doing CodeDeploy to ECS Fargate a few weeks ago for a side project and IMO it was overly complex.
  - 015a 4y ago
    Yup. You can get pretty dang far with just "the task definition is in cloudformation, so update the image in cloudformation and submit the template". For a bit more complexity, add ALB routing weights in, and the case for CodeDeploy is kind of weak. My statement that ECS needs it was probably overzealous; for most situations it doesn't, and if there's some crazy functionality it does I'm not aware of, you'd probably need something else in k8s as well (e.g. istio).

015a 4y ago

Right; and I don't feel that's a knock against K8s; its just the trade-offs it decided to make. A true Google-scale solution would be far worse to use, you can be sure of that. K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there). All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward. Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.

kuschku 4y ago
I'm using k8s even for my personal 2 node cluster. It's just so convenient to be able to use all the automation tools. I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc. At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.

arunc 4y ago

Curious how you ran 36k node cluster. Did it involve elastic scaling, etc? What are the alternatives for k8s?

KaiserPro 4y ago
Home grown: https://www.semanticscholar.org/paper/Robust-large-scale-rendering%3A-the-FQ-renderfarm-Hills-Vanns/6102a9bf5f3e54a98c87bb291bc22b96f20c2c83 https://www.semanticscholar.org/paper/Robust-large-scale-ren... off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alfred_Getting_Started.html https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf... although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high https://renderman.pixar.com/tractor https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/ https://www.opencue.io/ but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s. These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough. Nowadays, I'd use AWS batch, or at a push airflow.

singron 4y ago

Google's Borg clusters don't have wildly larger sizes than that despite having more years of development and a lot of motivation at that scale. They instead have a lot of clusters and transparent inter-cluster network topologies (i.e. you can choose clusters with very high bandwidths to each other). The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.