5 ms·

When we implemented Kubernetes at my employer, one of the things we made clear from the outset was that there is no LTS release of Kubernetes. Keeping k8s and f

When we implemented Kubernetes at my employer, one of the things we made clear from the outset was that there is no LTS release of Kubernetes. Keeping k8s and friends up to date is at least one full time job (ideally a half-pizza team). This also heavily influenced our choice of OS away from ones that emphasized LTS package management and towards more recent kernels with better BPF and kTLS support. (We also intentionally used short-lived TLS certificates to prevent anyone from running a cluster longer than several months without updates. Worked great. There are workarounds of course but anyone aware of them is the kind of person who updates anyway.)

polskibus 6y ago
What distro did you end up with?
- dharmab 6y ago
  CoreOS, then Flatcar.
  - polskibus 6y ago
    Did you have a look at Fedora CoreOS ?
zimbatm 6y ago
That's why k8s is not a good fit for early startups. It will consume too much of your time that is better spent developing a product that the public wants.
- AlphaSite 6y ago
  In that case use a managed cluster and it’s less work.
  - speedgoose 6y ago
    It has about the same issues but because it's managed and because you are a small company, nobody cares about your issues and you can do nothing about it.
  - zimbatm 6y ago
    You still have to do upgrades every 3 months, sometimes with breaking changes. Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.
    - q3k 6y ago
      Kubernetes clusters can be upgraded in place, though? GKE definitely allows you to do that.
      - zimbatm 6y ago
        It works most of the time. I have also seen multiple in-place upgrades ending up in weird places. Or some resource having changed API and we had to rush to upgrade the code. What will you do if that happens? It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.
      - dharmab 6y ago
        Yes, if you’re careful and do your research. Once you start extending Kubernetes you can back yourself into corners. We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).
  - capableweb 6y ago
    zimbatm mentioned "early startups", I take that to assume you've just started the company and you're working on getting the product market fit right. There is absolutely no need for any clusters for that, unless you're anticipating Google-level traffic on launch day. Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.
- znpy 6y ago
  imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course). another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far. We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far. (disclaimer: i'm not employed by sighup, just pleased to work with them)
  - zimbatm 6y ago
    Agreed. That's exactly what I have done with a number of startups. In one or two weeks, they get a full CI/CD pipeline that allows them to be productive. All of them are very happy with the result. It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production. If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.
  - mwcampbell 6y ago
    > baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?
    - znpy 6y ago
      That was meant to be an example really, to aim at something less automatic and complex like kubernetes but still automated. You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest. Now you can rollout your autoscaling groups without running packer and ansible.
amq 6y ago
Running several production clusters on GKE, I can say that it is definitely not a full-time job to keep it up too date and healthy in that setting. I would imagine it's a completely different story on-prem.
d33 6y ago
> half-pizza team Could you define this term? I tried Googling, but it looks like there are just two references and neither is clear to me.
- snarfy 6y ago
  When you buy food for the meeting, how many pizzas do you need? If it's one pizza that's a one pizza team.
  - oblio 6y ago
    It's also a specifically American reference. A 1-pizza team in most of Europe, including Italy, would be a 2, max 3 person team. Often it would be a 1 person team :-)
    - marcinzm 6y ago
      Same in parts of America but for very different reasons.
    - mrighele 6y ago
      In Italy a 1-pizza teams is absolutely a single person. We don't share pizzas :-)
- pseudalopex 6y ago
  2 or 3 people. It's a reference to Amazon's 2 pizza teams. The idea is your team is too big if 2 large pizzas isn't enough to feed everyone.
dimitar 6y ago
Do you use managed Kubernetes or kubeadm? I think if one uses a managed Kubernetes keeping it up-to-date is pretty easy. I did an AKS update in 10 minutes.
- dharmab 6y ago
  Neither. Both the managed offerings and kubeadm don’t meet our requirements. (Largely related to security, compliance, scale and multi-tenancy.)

polskibus 6y ago

What distro did you end up with?

dharmab 6y ago
CoreOS, then Flatcar.
- polskibus 6y ago
  Did you have a look at Fedora CoreOS ?

zimbatm 6y ago

That's why k8s is not a good fit for early startups. It will consume too much of your time that is better spent developing a product that the public wants.

AlphaSite 6y ago
In that case use a managed cluster and it’s less work.
- speedgoose 6y ago
  It has about the same issues but because it's managed and because you are a small company, nobody cares about your issues and you can do nothing about it.
- zimbatm 6y ago
  You still have to do upgrades every 3 months, sometimes with breaking changes. Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.
  - q3k 6y ago
    Kubernetes clusters can be upgraded in place, though? GKE definitely allows you to do that.
    - zimbatm 6y ago
      It works most of the time. I have also seen multiple in-place upgrades ending up in weird places. Or some resource having changed API and we had to rush to upgrade the code. What will you do if that happens? It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.
    - dharmab 6y ago
      Yes, if you’re careful and do your research. Once you start extending Kubernetes you can back yourself into corners. We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).
- capableweb 6y ago
  zimbatm mentioned "early startups", I take that to assume you've just started the company and you're working on getting the product market fit right. There is absolutely no need for any clusters for that, unless you're anticipating Google-level traffic on launch day. Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.
znpy 6y ago
imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course). another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far. We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far. (disclaimer: i'm not employed by sighup, just pleased to work with them)
- zimbatm 6y ago
  Agreed. That's exactly what I have done with a number of startups. In one or two weeks, they get a full CI/CD pipeline that allows them to be productive. All of them are very happy with the result. It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production. If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.
- mwcampbell 6y ago
  > baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?
  - znpy 6y ago
    That was meant to be an example really, to aim at something less automatic and complex like kubernetes but still automated. You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest. Now you can rollout your autoscaling groups without running packer and ansible.

amq 6y ago

Running several production clusters on GKE, I can say that it is definitely not a full-time job to keep it up too date and healthy in that setting. I would imagine it's a completely different story on-prem.

d33 6y ago

> half-pizza team Could you define this term? I tried Googling, but it looks like there are just two references and neither is clear to me.

snarfy 6y ago
When you buy food for the meeting, how many pizzas do you need? If it's one pizza that's a one pizza team.
- oblio 6y ago
  It's also a specifically American reference. A 1-pizza team in most of Europe, including Italy, would be a 2, max 3 person team. Often it would be a 1 person team :-)
  - marcinzm 6y ago
    Same in parts of America but for very different reasons.
  - mrighele 6y ago
    In Italy a 1-pizza teams is absolutely a single person. We don't share pizzas :-)
pseudalopex 6y ago
2 or 3 people. It's a reference to Amazon's 2 pizza teams. The idea is your team is too big if 2 large pizzas isn't enough to feed everyone.

dimitar 6y ago

Do you use managed Kubernetes or kubeadm? I think if one uses a managed Kubernetes keeping it up-to-date is pretty easy. I did an AKS update in 10 minutes.

dharmab 6y ago
Neither. Both the managed offerings and kubeadm don’t meet our requirements. (Largely related to security, compliance, scale and multi-tenancy.)