5 ms·

> Kubernetes is up to 80% cheaper compared to PaaS like Heroku What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. Th

> Kubernetes is up to 80% cheaper compared to PaaS like Heroku What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.

danpalmer 3y ago
I would guess "up to" means once these costs you mention are amortized. Sure k8s isn't going to beat a $10 dyno or however much they cost now, but I bet a cluster of 10 large VMs each running 50 containers blows the equivalent Heroku setup out of the water on costs. K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
jiggawatts 3y ago
For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences. The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!) At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
- dividedbyzero 3y ago
  > For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?
  - jiggawatts 3y ago
    A few months back it was a headline statement on the kubernetes.io web site. Something like 'XX% of organisations with more than 1,000 developers use Kubernetes". Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
    - sanitycheck 3y ago
      "XX% of organisations with more than 1,000 developers use Kubernetes" does not mean "most Kubernetes users work for organisations with more than 1,000 developers". In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
      - smashed 3y ago
        Heroku had their share of problems recently no? It seems mismanaged and on the way down, sadly..? And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really? The real alternative is a Linux VM with docker-compose. Once you outgrow that, managed kubernetes is the logical next step, even for a small team.
- ekidd 3y ago
  > For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services. I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically. Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them. I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong. However, I do have several rules for Kubernetes happiness: 1. If your app is simple enough to run on a Heroku-like service, do that instead. 2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book. 3. Pay a cloud vendor for managed Kubernetes. 4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays. I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
  - bavell 3y ago
    As a solopreneur using tiny k8s clusters, I agree wholeheartedly with #3 and #4 of your list. I don't think #2 is necessary and #1 is probably a good choice if you haven't already jumped onto the k8s bandwagon.
- bavell 3y ago
  > At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
  - alemanek 3y ago
    This is interesting. I have only used k8s and k3s at the larger scales. For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
- crabbone 3y ago
  KVM automatically overprovisions memory. Probably other hypervisors do it too. The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y). I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure. The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.
jarym 3y ago
I’ve been using Civo for a while to get round some of the GCP costs (which are as bad as AWS for indie devs). So far no complaints and I like that it feels faster than GCP.
bavell 3y ago
Idk what the author is referring to specifically but I can speak to my experience. I've been running a few single-node k8s clusters for a few years now. On GCP, the control plane fee is waived if you're running in a single region (no $70/mo fee). So basically you only pay for the VM itself. As a solopreneur, I've saved so much time and money letting k8s run my workloads while I rack up billable hours and build products.
- jbnorth 3y ago
  Why even use kubernetes then? You’d be better off just deploying the applications with docker compose on the host.
  - mplewis 3y ago
    Kubernetes makes it reasonably painless to wire up things like ingress and supports nice features such as operators.
    - benterix 3y ago
      I'd argue ingress is even easier with Docker Compose. As for operators, it depends, but e.g. database operators are not really necessary on a single node cluster as you don't have discrepancies between nodes.
    - smashed 3y ago
      I like the fact you qualified ingress as "reasonably" painless. Frankly it is not painless at all. Ingress is complicated and there is a ton of way to configure it, with all the forced tie in with the cloud provider's offering, be it l3 or l4 load balancing, various proxy services, etc. It's not easy at all either on docker-compose. You have to roll your own proxy container and take care of TLS termination, cert renewals, manage specific headers and config, etc. I am not sure what exactly I want to add to the conversation, only that either way, it's not "easy". I'd be glad to be proven wrong if you have a specific solution that makes ingress really painless.
mplewis 3y ago
You can run a single-node cluster for $5 on DigitalOcean, where a single Heroku prod node would cost $25. This price estimate doesn’t seem outlandish to me, but you do have to push it a bit to get to 80%.
- benterix 3y ago
  Sure, but if for some reason you decide to run a single-node cluster in production, you'd probably be better off using Docker Compose.
  - dividedbyzero 3y ago
    If everyone else runs Kubernetes and no one uses docker-compose you might be looking at a ton of overhead, e.g. ITSEC may still expect you to run all of the stuff they offer a helm chart for. There may be no easy way to get the level of access on a VM to install docker-compose in the first place. Might well be worth running a single-node cluster just to be able to use an existing platform.
  - mplewis 3y ago
    I can't buy a managed Docker Compose system from DigitalOcean, so this would be harder to get started with than managed k8s.
deleted 3y ago
[deleted]
pnocera 3y ago
I'm running a decent k3s 4 nodes cluster + 1 VM for the database at Hetzner for a total cost of 88€/month. It's not that expensive.
- benterix 3y ago
  What instance sizes? ARM or Intel?
mrweasel 3y ago
So my take is that if you're concerned with the expense of 3+ VMs, then Kubernetes isn't for you. We simply run the worker nodes on physical hardware, anything below that is a little weird for production environments, in my opinion. Something like k3s makes a lot of sense for development environments though. It is much easier to create a production like environment with something like k3s than it ever was using VMs. I do feel like there's room in the middle for something more flexible than Ansible and Puppet, but less complex than Kubernetes. If you're just running a few VMs, "upgrading" to Kubernetes doesn't seem like it's worth the hassle. On the other hand, deploying the same application, or applications to say 4 or 8 VMs, that starting to get i little annoying. I'd argue for a tool that augments Docker-Compose, one that understands distribution of applications, but doesn't deal with networking (the network team can handle that). I suppose a little like Nomad?

danpalmer 3y ago

I would guess "up to" means once these costs you mention are amortized. Sure k8s isn't going to beat a $10 dyno or however much they cost now, but I bet a cluster of 10 large VMs each running 50 containers blows the equivalent Heroku setup out of the water on costs. K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.

jiggawatts 3y ago

For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences. The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!) At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.

dividedbyzero 3y ago
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?
- jiggawatts 3y ago
  A few months back it was a headline statement on the kubernetes.io web site. Something like 'XX% of organisations with more than 1,000 developers use Kubernetes". Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
  - sanitycheck 3y ago
    "XX% of organisations with more than 1,000 developers use Kubernetes" does not mean "most Kubernetes users work for organisations with more than 1,000 developers". In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
    - smashed 3y ago
      Heroku had their share of problems recently no? It seems mismanaged and on the way down, sadly..? And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really? The real alternative is a Linux VM with docker-compose. Once you outgrow that, managed kubernetes is the logical next step, even for a small team.
ekidd 3y ago
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services. I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically. Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them. I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong. However, I do have several rules for Kubernetes happiness: 1. If your app is simple enough to run on a Heroku-like service, do that instead. 2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book. 3. Pay a cloud vendor for managed Kubernetes. 4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays. I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
- bavell 3y ago
  As a solopreneur using tiny k8s clusters, I agree wholeheartedly with #3 and #4 of your list. I don't think #2 is necessary and #1 is probably a good choice if you haven't already jumped onto the k8s bandwagon.
bavell 3y ago
> At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
- alemanek 3y ago
  This is interesting. I have only used k8s and k3s at the larger scales. For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
crabbone 3y ago
KVM automatically overprovisions memory. Probably other hypervisors do it too. The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y). I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure. The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.

jarym 3y ago

I’ve been using Civo for a while to get round some of the GCP costs (which are as bad as AWS for indie devs). So far no complaints and I like that it feels faster than GCP.

bavell 3y ago

Idk what the author is referring to specifically but I can speak to my experience. I've been running a few single-node k8s clusters for a few years now. On GCP, the control plane fee is waived if you're running in a single region (no $70/mo fee). So basically you only pay for the VM itself. As a solopreneur, I've saved so much time and money letting k8s run my workloads while I rack up billable hours and build products.

jbnorth 3y ago
Why even use kubernetes then? You’d be better off just deploying the applications with docker compose on the host.
- mplewis 3y ago
  Kubernetes makes it reasonably painless to wire up things like ingress and supports nice features such as operators.
  - benterix 3y ago
    I'd argue ingress is even easier with Docker Compose. As for operators, it depends, but e.g. database operators are not really necessary on a single node cluster as you don't have discrepancies between nodes.
  - smashed 3y ago
    I like the fact you qualified ingress as "reasonably" painless. Frankly it is not painless at all. Ingress is complicated and there is a ton of way to configure it, with all the forced tie in with the cloud provider's offering, be it l3 or l4 load balancing, various proxy services, etc. It's not easy at all either on docker-compose. You have to roll your own proxy container and take care of TLS termination, cert renewals, manage specific headers and config, etc. I am not sure what exactly I want to add to the conversation, only that either way, it's not "easy". I'd be glad to be proven wrong if you have a specific solution that makes ingress really painless.

mplewis 3y ago

You can run a single-node cluster for $5 on DigitalOcean, where a single Heroku prod node would cost $25. This price estimate doesn’t seem outlandish to me, but you do have to push it a bit to get to 80%.

benterix 3y ago
Sure, but if for some reason you decide to run a single-node cluster in production, you'd probably be better off using Docker Compose.
- dividedbyzero 3y ago
  If everyone else runs Kubernetes and no one uses docker-compose you might be looking at a ton of overhead, e.g. ITSEC may still expect you to run all of the stuff they offer a helm chart for. There may be no easy way to get the level of access on a VM to install docker-compose in the first place. Might well be worth running a single-node cluster just to be able to use an existing platform.
- mplewis 3y ago
  I can't buy a managed Docker Compose system from DigitalOcean, so this would be harder to get started with than managed k8s.

deleted 3y ago

[deleted]

pnocera 3y ago

I'm running a decent k3s 4 nodes cluster + 1 VM for the database at Hetzner for a total cost of 88€/month. It's not that expensive.

benterix 3y ago
What instance sizes? ARM or Intel?

mrweasel 3y ago

So my take is that if you're concerned with the expense of 3+ VMs, then Kubernetes isn't for you. We simply run the worker nodes on physical hardware, anything below that is a little weird for production environments, in my opinion. Something like k3s makes a lot of sense for development environments though. It is much easier to create a production like environment with something like k3s than it ever was using VMs. I do feel like there's room in the middle for something more flexible than Ansible and Puppet, but less complex than Kubernetes. If you're just running a few VMs, "upgrading" to Kubernetes doesn't seem like it's worth the hassle. On the other hand, deploying the same application, or applications to say 4 or 8 VMs, that starting to get i little annoying. I'd argue for a tool that augments Docker-Compose, one that understands distribution of applications, but doesn't deal with networking (the network team can handle that). I suppose a little like Nomad?