4 ms·

Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?

by oauea 6y ago

majormajor 6y ago
Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?
- oauea 6y ago
  People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.
- briffle 6y ago
  I use Kubernetes for work. We utilize GKE in google cloud. There have been a few times things have required some editing of our YAML files, such as when we had to change the first line of our yaml from: apiVersion: apps/Betav1 kind: Deployment to: apiVersion: apps/v1 kind: Deployment Because in K8s 1.16, they stopped the backwards compatibility for Beta on deployments. But that was quite some time they left it in for backwards compatibility, and it was a simple sed script to change it. We have never changed our Dockerfiles except to change the FROM line when we update a base image. They often introduce new features, but try to keep things as stable as possible for existing setups. In our case, we store our YAML in git, and can deploy a new cluster with all our microservices in about 10 min (most of that is delays in the google global load-balancer setting up a TLS certificate for us, the machines are up and running after just a few minutes). There can be a bit more scripting needed to do updates of the nodes to newer builds with no downtime, if your pods aren't totally stateless, but compared to what it used to be at an old job, with C apps running on Linux, behind load-balancers that had to be removed, upgraded, and added, a rolling update is like black magic.
  - Speranskiy 6y ago
    > and can deploy a new cluster with all our microservices in about 10 min I know nothing about k8s yet, but try to solve the same issue with docker swarm and one thing I'm unable to get my head around is this: as soon as yaml files are stored in apps' repos and deploy is being initiated by commiting to them, how do you let the repos know about the new deployment? Or is there a tool that is present in k8s but is missing in Docker Swarm that knows about scattered yaml files' locations (git repos) and pulls them on the new deployment?
  - necovek 6y ago
    > if your pods aren't totally stateless This is exactly the bit I care about: what do I do with "stateful" systems (my db storage, mail server data,...)? If I just mount those from external volumes, I lose a lot of idempotency and I now have to worry about whether I am trying to attach that volume to PG 9.3.1 or 9.3.2 or 12.0 (some are fine, others might cause data corruption). I know idempotent deploys are all the rage, but all those deployed apps are there to serve some data which is as stateful as it can be. I'd like a system where those external volumes are automatic snapshots on LVM (or ZFS/btrfs) when attached, but that introduces a whole another level of what-now if you need to go back to an older, now slightly stale data set. How does k8s solve this problem for me?
    - tasuki 6y ago
      This question always gets asked. I'm hoping for an answer one day too!
- audleman 6y ago
  The people claiming it's easy are, well, lying. Our deployments fail all the time for things like Google moved their Helm images to a new repo. Kubernetes is still early tech and thus has more moving parts than a more well established stack. Keeping up to date is crucial and no small feat.
fluential 6y ago
Just use https://github.com/fluxcd/flux https://github.com/fluxcd/flux — „The GitOps Kubernetes operator”
bayindirh 6y ago
In my case, it's 15 minutes. 1. Get a new server 2. Connect ethernet and IPMI ports 3. Add node definition to XCAT 4. Set boot to network, arm node for installation, power-on 5. Profit. 6. Get a coffee. It's 15 minutes for [1-150] servers. Slightly longer for 150+. Because, network. Any further setup is done via Salt, if necessary. With a single state file. Fun fact: I provision K8S nodes that way too.
spondyl 6y ago
At least the first 10 minutes is waiting for my corporate laptop to boot up ;)

Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?

by oauea 6y ago

Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?

Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?

oauea 6y ago
People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.
briffle 6y ago
I use Kubernetes for work. We utilize GKE in google cloud. There have been a few times things have required some editing of our YAML files, such as when we had to change the first line of our yaml from: apiVersion: apps/Betav1 kind: Deployment to: apiVersion: apps/v1 kind: Deployment Because in K8s 1.16, they stopped the backwards compatibility for Beta on deployments. But that was quite some time they left it in for backwards compatibility, and it was a simple sed script to change it. We have never changed our Dockerfiles except to change the FROM line when we update a base image. They often introduce new features, but try to keep things as stable as possible for existing setups. In our case, we store our YAML in git, and can deploy a new cluster with all our microservices in about 10 min (most of that is delays in the google global load-balancer setting up a TLS certificate for us, the machines are up and running after just a few minutes). There can be a bit more scripting needed to do updates of the nodes to newer builds with no downtime, if your pods aren't totally stateless, but compared to what it used to be at an old job, with C apps running on Linux, behind load-balancers that had to be removed, upgraded, and added, a rolling update is like black magic.
- Speranskiy 6y ago
  > and can deploy a new cluster with all our microservices in about 10 min I know nothing about k8s yet, but try to solve the same issue with docker swarm and one thing I'm unable to get my head around is this: as soon as yaml files are stored in apps' repos and deploy is being initiated by commiting to them, how do you let the repos know about the new deployment? Or is there a tool that is present in k8s but is missing in Docker Swarm that knows about scattered yaml files' locations (git repos) and pulls them on the new deployment?
- necovek 6y ago
  > if your pods aren't totally stateless This is exactly the bit I care about: what do I do with "stateful" systems (my db storage, mail server data,...)? If I just mount those from external volumes, I lose a lot of idempotency and I now have to worry about whether I am trying to attach that volume to PG 9.3.1 or 9.3.2 or 12.0 (some are fine, others might cause data corruption). I know idempotent deploys are all the rage, but all those deployed apps are there to serve some data which is as stateful as it can be. I'd like a system where those external volumes are automatic snapshots on LVM (or ZFS/btrfs) when attached, but that introduces a whole another level of what-now if you need to go back to an older, now slightly stale data set. How does k8s solve this problem for me?
  - tasuki 6y ago
    This question always gets asked. I'm hoping for an answer one day too!
audleman 6y ago
The people claiming it's easy are, well, lying. Our deployments fail all the time for things like Google moved their Helm images to a new repo. Kubernetes is still early tech and thus has more moving parts than a more well established stack. Keeping up to date is crucial and no small feat.

fluential 6y ago

Just use https://github.com/fluxcd/flux https://github.com/fluxcd/flux — „The GitOps Kubernetes operator”

bayindirh 6y ago

In my case, it's 15 minutes. 1. Get a new server 2. Connect ethernet and IPMI ports 3. Add node definition to XCAT 4. Set boot to network, arm node for installation, power-on 5. Profit. 6. Get a coffee. It's 15 minutes for [1-150] servers. Slightly longer for 150+. Because, network. Any further setup is done via Salt, if necessary. With a single state file. Fun fact: I provision K8S nodes that way too.

spondyl 6y ago

At least the first 10 minutes is waiting for my corporate laptop to boot up ;)