6 ms·

> What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command lin

> What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Can you expand on how this is different from traditional networking? I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network, and it wasn't like I had to keep tinkering with it. Once I had network storage, I could use it from lots of different things, etc. But eventually you need software patches, updates, etc, and that was where the ongoing pain would sometimes crop up. Is that so different here? From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort.

oauea 6y ago
Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?
- majormajor 6y ago
  Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?
  - oauea 6y ago
    People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.
  - briffle 6y ago
    I use Kubernetes for work. We utilize GKE in google cloud. There have been a few times things have required some editing of our YAML files, such as when we had to change the first line of our yaml from: apiVersion: apps/Betav1 kind: Deployment to: apiVersion: apps/v1 kind: Deployment Because in K8s 1.16, they stopped the backwards compatibility for Beta on deployments. But that was quite some time they left it in for backwards compatibility, and it was a simple sed script to change it. We have never changed our Dockerfiles except to change the FROM line when we update a base image. They often introduce new features, but try to keep things as stable as possible for existing setups. In our case, we store our YAML in git, and can deploy a new cluster with all our microservices in about 10 min (most of that is delays in the google global load-balancer setting up a TLS certificate for us, the machines are up and running after just a few minutes). There can be a bit more scripting needed to do updates of the nodes to newer builds with no downtime, if your pods aren't totally stateless, but compared to what it used to be at an old job, with C apps running on Linux, behind load-balancers that had to be removed, upgraded, and added, a rolling update is like black magic.
    - Speranskiy 6y ago
      > and can deploy a new cluster with all our microservices in about 10 min I know nothing about k8s yet, but try to solve the same issue with docker swarm and one thing I'm unable to get my head around is this: as soon as yaml files are stored in apps' repos and deploy is being initiated by commiting to them, how do you let the repos know about the new deployment? Or is there a tool that is present in k8s but is missing in Docker Swarm that knows about scattered yaml files' locations (git repos) and pulls them on the new deployment?
    - necovek 6y ago
      > if your pods aren't totally stateless This is exactly the bit I care about: what do I do with "stateful" systems (my db storage, mail server data,...)? If I just mount those from external volumes, I lose a lot of idempotency and I now have to worry about whether I am trying to attach that volume to PG 9.3.1 or 9.3.2 or 12.0 (some are fine, others might cause data corruption). I know idempotent deploys are all the rage, but all those deployed apps are there to serve some data which is as stateful as it can be. I'd like a system where those external volumes are automatic snapshots on LVM (or ZFS/btrfs) when attached, but that introduces a whole another level of what-now if you need to go back to an older, now slightly stale data set. How does k8s solve this problem for me?
  - audleman 6y ago
    The people claiming it's easy are, well, lying. Our deployments fail all the time for things like Google moved their Helm images to a new repo. Kubernetes is still early tech and thus has more moving parts than a more well established stack. Keeping up to date is crucial and no small feat.
- fluential 6y ago
  Just use https://github.com/fluxcd/flux https://github.com/fluxcd/flux — „The GitOps Kubernetes operator”
- bayindirh 6y ago
  In my case, it's 15 minutes. 1. Get a new server 2. Connect ethernet and IPMI ports 3. Add node definition to XCAT 4. Set boot to network, arm node for installation, power-on 5. Profit. 6. Get a coffee. It's 15 minutes for [1-150] servers. Slightly longer for 150+. Because, network. Any further setup is done via Salt, if necessary. With a single state file. Fun fact: I provision K8S nodes that way too.
- spondyl 6y ago
  At least the first 10 minutes is waiting for my corporate laptop to boot up ;)
bryanlarsen 6y ago
The big difference is that kubernetes YAML is in a change control system, and specifies what you want rather than how you get it.
vinceguidry 6y ago
What's different is that the applications and the infrastructure are all managed and stored in the same location. There's a lot less you have to keep track of. Sure, you can get it all done and have it stable the traditional way, but k8s makes it fun!
acoard 6y ago
>Can you expand on how this is different from traditional networking? Not OP, but I can chime in here. It's not that it's solved in this specific deployment, but it's solved in yaml files that you can re-deploy to any environment. Once I get my bundle of yaml files, I can just point it to any cluster (or namespace) et voila, it'll be deployed. It's "infrastructure-as-code"[0]. To be clear, Kubernetes doesn't have a monopoly on this approach, but it certainly follows it. Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink. >I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network >From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort. You're not wrong here though. Sometimes there are maintenance tasks you have to do on the whole cluster (make sure there's enough storage, everything is updated, certs are good, etc). But this is orthogonal to having your infrastructure-as-code in yaml files. The yaml files assume a healthy cluster/namespace, but besides that any cluster should be fungible. This is treating your servers like cattle, instead of pets. [0] https://docs.microsoft.com/en-us/azure/devops/learn/what-is-infrastructure-as-code https://docs.microsoft.com/en-us/azure/devops/learn/what-is-...
- WanderPanda 6y ago
  > Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink. Very poetic, I love it! These kind of internet conventional wisdoms are gold, I also like: Batteries included but swappable Free as in Beer vs. free as in speech Are there other contenders?
nonameiguess 6y ago
There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc. Maybe, if you're lucky, you had some set of really good and reliable Sys Admins that figured out a robust way to script and configure the setup process of your original on-premise data center and they captured that in very good, well-maintained documentation. If you're even luckier, maybe those same guys still work for you. If not, well, Kubernetes provides an open standard that a lot of different people know how to use and the documentation for how everything is setup is directly in your source control. There's nothing exactly unique about this, but it is an actual open standard. It hasn't been owned by Google for years. It's owned by a non-profit. It's open source. If your deployments work on a cluster provided by one Kubernetes engine implementation, it'll work on any of them. You can roll your own, use GKE, EKS, whatever Microsoft offers, and it'll work in exactly the same way wherever you go. For one person, this means nothing. This guy is just doing it for fun. But for organizations that used to lose millions of dollars a year and sometimes their entire market position thanks to vendor lock in or had to experience months of downtime for a data center migration, they may find something like this useful. Or maybe you just don't want to rely too heavily on Dennis Nedry and want to be able to bring in anyone with a couple years of experience using an open source, open standard toolset who can come in and be reasonably expected to understand how your system works pretty quickly without needing ten years of tribal knowledge.
- politician 6y ago
  > There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc. It sounds like you're making the argument that deploying k8s for a startup is an extreme case of premature optimization.
  - anthonyskipper 6y ago
    Nah, it's par for course for startups for the following reasons... You can get tons of credits for your startup, typically hundreds of thousands from Microsoft, Amazon, etc. -- Eventually you run out of credits, so you switch providers. I did this 3 times at a startup and got three years of free infrastructure. If you are selling enterprise software, then you use k8s so you can deploy at enterprise without having to integrate with all the wacky requirements. You can build a saas that offers isolated service nodes on k8s infra pretty easy if you just give everyone a small vm with a k8s cluster on it. If you use a scale to zero model, your infra costs are a lot cheaper. Simiarly the auto-scaling k8s capabilities are really nice and it's awesome to be able to easily build systems that can scale up massively. You never know when you r startup will get popular.
ytjohn 6y ago
From a sysadmin/config management side of things, and since I "dabble" with networking, I've always been rather jealous of the network appliances. All the serious network devices had a single configuration file. Often the lines in the config file were equivalent to shell commands. For any network device, you could put the list of all the commands in notepad, as a script. But, if you exported the config file, it would look like a nicer version of your "script". With config management, be it puppet, chef, salt, ansible, you start to approach that. Except these only manage the resources you define. If your config management manages apache resources, and someone wandered in and installed nginx that's out of scope. If you've got a list of machine-local usernames that you want to exist, you might not thinking about ensuring that other usernames are absent. With kubernetes, it gets a bit closer. Let me preface that standing and running your own kubernetes cluster is a separate level of complexity, but once you have a k8s cluster, you can dump out a list of all resources (namespaces, pods, volumeclaims, ingress controllers, services...), modify that resulting file and apply it. Most people work in chunks, having a yaml file for each set of associated resources, or template those into a helm chart. With that, you can put all associated resources into a release, which you can delete and recreate as you desire. There's still lots of gotchas, especially around persistence. It's not uncommon to find an orphaned volume claim that prevents your release upgrade from continuing until you dig in with 3 coworkers watching you as you try to remember if it's safe to delete that or not.

oauea 6y ago

Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?

majormajor 6y ago
Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?
- oauea 6y ago
  People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.
- briffle 6y ago
  I use Kubernetes for work. We utilize GKE in google cloud. There have been a few times things have required some editing of our YAML files, such as when we had to change the first line of our yaml from: apiVersion: apps/Betav1 kind: Deployment to: apiVersion: apps/v1 kind: Deployment Because in K8s 1.16, they stopped the backwards compatibility for Beta on deployments. But that was quite some time they left it in for backwards compatibility, and it was a simple sed script to change it. We have never changed our Dockerfiles except to change the FROM line when we update a base image. They often introduce new features, but try to keep things as stable as possible for existing setups. In our case, we store our YAML in git, and can deploy a new cluster with all our microservices in about 10 min (most of that is delays in the google global load-balancer setting up a TLS certificate for us, the machines are up and running after just a few minutes). There can be a bit more scripting needed to do updates of the nodes to newer builds with no downtime, if your pods aren't totally stateless, but compared to what it used to be at an old job, with C apps running on Linux, behind load-balancers that had to be removed, upgraded, and added, a rolling update is like black magic.
  - Speranskiy 6y ago
    > and can deploy a new cluster with all our microservices in about 10 min I know nothing about k8s yet, but try to solve the same issue with docker swarm and one thing I'm unable to get my head around is this: as soon as yaml files are stored in apps' repos and deploy is being initiated by commiting to them, how do you let the repos know about the new deployment? Or is there a tool that is present in k8s but is missing in Docker Swarm that knows about scattered yaml files' locations (git repos) and pulls them on the new deployment?
  - necovek 6y ago
    > if your pods aren't totally stateless This is exactly the bit I care about: what do I do with "stateful" systems (my db storage, mail server data,...)? If I just mount those from external volumes, I lose a lot of idempotency and I now have to worry about whether I am trying to attach that volume to PG 9.3.1 or 9.3.2 or 12.0 (some are fine, others might cause data corruption). I know idempotent deploys are all the rage, but all those deployed apps are there to serve some data which is as stateful as it can be. I'd like a system where those external volumes are automatic snapshots on LVM (or ZFS/btrfs) when attached, but that introduces a whole another level of what-now if you need to go back to an older, now slightly stale data set. How does k8s solve this problem for me?
- audleman 6y ago
  The people claiming it's easy are, well, lying. Our deployments fail all the time for things like Google moved their Helm images to a new repo. Kubernetes is still early tech and thus has more moving parts than a more well established stack. Keeping up to date is crucial and no small feat.
fluential 6y ago
Just use https://github.com/fluxcd/flux https://github.com/fluxcd/flux — „The GitOps Kubernetes operator”
bayindirh 6y ago
In my case, it's 15 minutes. 1. Get a new server 2. Connect ethernet and IPMI ports 3. Add node definition to XCAT 4. Set boot to network, arm node for installation, power-on 5. Profit. 6. Get a coffee. It's 15 minutes for [1-150] servers. Slightly longer for 150+. Because, network. Any further setup is done via Salt, if necessary. With a single state file. Fun fact: I provision K8S nodes that way too.
spondyl 6y ago
At least the first 10 minutes is waiting for my corporate laptop to boot up ;)

bryanlarsen 6y ago

The big difference is that kubernetes YAML is in a change control system, and specifies what you want rather than how you get it.

vinceguidry 6y ago

What's different is that the applications and the infrastructure are all managed and stored in the same location. There's a lot less you have to keep track of. Sure, you can get it all done and have it stable the traditional way, but k8s makes it fun!

acoard 6y ago

>Can you expand on how this is different from traditional networking? Not OP, but I can chime in here. It's not that it's solved in this specific deployment, but it's solved in yaml files that you can re-deploy to any environment. Once I get my bundle of yaml files, I can just point it to any cluster (or namespace) et voila, it'll be deployed. It's "infrastructure-as-code"[0]. To be clear, Kubernetes doesn't have a monopoly on this approach, but it certainly follows it. Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink. >I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network >From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort. You're not wrong here though. Sometimes there are maintenance tasks you have to do on the whole cluster (make sure there's enough storage, everything is updated, certs are good, etc). But this is orthogonal to having your infrastructure-as-code in yaml files. The yaml files assume a healthy cluster/namespace, but besides that any cluster should be fungible. This is treating your servers like cattle, instead of pets. [0] https://docs.microsoft.com/en-us/azure/devops/learn/what-is-infrastructure-as-code https://docs.microsoft.com/en-us/azure/devops/learn/what-is-...

WanderPanda 6y ago
> Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink. Very poetic, I love it! These kind of internet conventional wisdoms are gold, I also like: Batteries included but swappable Free as in Beer vs. free as in speech Are there other contenders?

nonameiguess 6y ago

There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc. Maybe, if you're lucky, you had some set of really good and reliable Sys Admins that figured out a robust way to script and configure the setup process of your original on-premise data center and they captured that in very good, well-maintained documentation. If you're even luckier, maybe those same guys still work for you. If not, well, Kubernetes provides an open standard that a lot of different people know how to use and the documentation for how everything is setup is directly in your source control. There's nothing exactly unique about this, but it is an actual open standard. It hasn't been owned by Google for years. It's owned by a non-profit. It's open source. If your deployments work on a cluster provided by one Kubernetes engine implementation, it'll work on any of them. You can roll your own, use GKE, EKS, whatever Microsoft offers, and it'll work in exactly the same way wherever you go. For one person, this means nothing. This guy is just doing it for fun. But for organizations that used to lose millions of dollars a year and sometimes their entire market position thanks to vendor lock in or had to experience months of downtime for a data center migration, they may find something like this useful. Or maybe you just don't want to rely too heavily on Dennis Nedry and want to be able to bring in anyone with a couple years of experience using an open source, open standard toolset who can come in and be reasonably expected to understand how your system works pretty quickly without needing ten years of tribal knowledge.

politician 6y ago
> There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc. It sounds like you're making the argument that deploying k8s for a startup is an extreme case of premature optimization.
- anthonyskipper 6y ago
  Nah, it's par for course for startups for the following reasons... You can get tons of credits for your startup, typically hundreds of thousands from Microsoft, Amazon, etc. -- Eventually you run out of credits, so you switch providers. I did this 3 times at a startup and got three years of free infrastructure. If you are selling enterprise software, then you use k8s so you can deploy at enterprise without having to integrate with all the wacky requirements. You can build a saas that offers isolated service nodes on k8s infra pretty easy if you just give everyone a small vm with a k8s cluster on it. If you use a scale to zero model, your infra costs are a lot cheaper. Simiarly the auto-scaling k8s capabilities are really nice and it's awesome to be able to easily build systems that can scale up massively. You never know when you r startup will get popular.

ytjohn 6y ago

From a sysadmin/config management side of things, and since I "dabble" with networking, I've always been rather jealous of the network appliances. All the serious network devices had a single configuration file. Often the lines in the config file were equivalent to shell commands. For any network device, you could put the list of all the commands in notepad, as a script. But, if you exported the config file, it would look like a nicer version of your "script". With config management, be it puppet, chef, salt, ansible, you start to approach that. Except these only manage the resources you define. If your config management manages apache resources, and someone wandered in and installed nginx that's out of scope. If you've got a list of machine-local usernames that you want to exist, you might not thinking about ensuring that other usernames are absent. With kubernetes, it gets a bit closer. Let me preface that standing and running your own kubernetes cluster is a separate level of complexity, but once you have a k8s cluster, you can dump out a list of all resources (namespaces, pods, volumeclaims, ingress controllers, services...), modify that resulting file and apply it. Most people work in chunks, having a yaml file for each set of associated resources, or template those into a helm chart. With that, you can put all associated resources into a release, which you can delete and recreate as you desire. There's still lots of gotchas, especially around persistence. It's not uncommon to find an orphaned volume claim that prevents your release upgrade from continuing until you dig in with 3 coworkers watching you as you try to remember if it's safe to delete that or not.