3 ms·

For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively a

For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences. The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!) At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.

dividedbyzero 3y ago
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?
- jiggawatts 3y ago
  A few months back it was a headline statement on the kubernetes.io web site. Something like 'XX% of organisations with more than 1,000 developers use Kubernetes". Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
  - sanitycheck 3y ago
    "XX% of organisations with more than 1,000 developers use Kubernetes" does not mean "most Kubernetes users work for organisations with more than 1,000 developers". In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
    - smashed 3y ago
      Heroku had their share of problems recently no? It seems mismanaged and on the way down, sadly..? And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really? The real alternative is a Linux VM with docker-compose. Once you outgrow that, managed kubernetes is the logical next step, even for a small team.
ekidd 3y ago
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services. I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically. Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them. I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong. However, I do have several rules for Kubernetes happiness: 1. If your app is simple enough to run on a Heroku-like service, do that instead. 2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book. 3. Pay a cloud vendor for managed Kubernetes. 4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays. I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
- bavell 3y ago
  As a solopreneur using tiny k8s clusters, I agree wholeheartedly with #3 and #4 of your list. I don't think #2 is necessary and #1 is probably a good choice if you haven't already jumped onto the k8s bandwagon.
bavell 3y ago
> At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
- alemanek 3y ago
  This is interesting. I have only used k8s and k3s at the larger scales. For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
crabbone 3y ago
KVM automatically overprovisions memory. Probably other hypervisors do it too. The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y). I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure. The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.

dividedbyzero 3y ago

> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?

jiggawatts 3y ago
A few months back it was a headline statement on the kubernetes.io web site. Something like 'XX% of organisations with more than 1,000 developers use Kubernetes". Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
- sanitycheck 3y ago
  "XX% of organisations with more than 1,000 developers use Kubernetes" does not mean "most Kubernetes users work for organisations with more than 1,000 developers". In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
  - smashed 3y ago
    Heroku had their share of problems recently no? It seems mismanaged and on the way down, sadly..? And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really? The real alternative is a Linux VM with docker-compose. Once you outgrow that, managed kubernetes is the logical next step, even for a small team.

ekidd 3y ago

> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services. I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically. Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them. I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong. However, I do have several rules for Kubernetes happiness: 1. If your app is simple enough to run on a Heroku-like service, do that instead. 2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book. 3. Pay a cloud vendor for managed Kubernetes. 4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays. I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.

bavell 3y ago
As a solopreneur using tiny k8s clusters, I agree wholeheartedly with #3 and #4 of your list. I don't think #2 is necessary and #1 is probably a good choice if you haven't already jumped onto the k8s bandwagon.

bavell 3y ago

> At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.

alemanek 3y ago
This is interesting. I have only used k8s and k3s at the larger scales. For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.

crabbone 3y ago

KVM automatically overprovisions memory. Probably other hypervisors do it too. The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y). I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure. The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.