5 ms·

I wouldn't say it's cargo-culting, but, it's definitely silly (and intended to be). Not having to manage systems anymore, and fully relying on kubernetes to co

I wouldn't say it's cargo-culting, but, it's definitely silly (and intended to be). Not having to manage systems anymore, and fully relying on kubernetes to configure and manage everything is great, especially leveraging the redundancy and reliability of it.

Alupis 2y ago
People so very often are quick to look down upon anything using k8s these days, often citing complexity and some hand-wavy statement about "you're not web scale" or similar. These statements are usually accompanied by complex suggestions as "more simple" alternatives. On one hand you can manually provision a VM, configure the VM's firewall, update the OS, install several packages, write a bash script which can be used with cron, configure cron, setup backups, and more, followed by routine maintenance that is required to keep this system running and secure. This dance has to be done every time you deploy the app... oh, and don't forget that hack you did one late night to get your failing app back online but forgot to document! Or, on the other hand, you can write 1-3 relatively simple yaml files, just once, that explicitly describe what your app should look like within the cluster, and then it just does it's thing forever. k8s is very flexible and very useful, even for simple things. Why would you not use it, should be the default question.
- drdaeman 2y ago
  Setting up and maintaining along the happy path is not an issue, it’s well-documented and has existing automations if desirable so you can set up a cluster in just a few minutes (if you have the experience already). And YAML files are nearly trivial in most cases. Boilerplate is not pretty but it’s easy. The only Problem (with a capital P) with K8s is when it suddenly and unexpectedly breaks down - something fails and you need to figure out what is going on. That’s where its complexity bites you, hard - because it’s a whole new OS on top of another OS - so, a whole new giant behemoth to learn and debug. If you’re lucky it’s already documented by someone so you can just follow the recipe, but if luck runs out (it always eventually does) it’s not a common case you’re going to have some very unpleasant time. I’m a lazy ass, I hate having to debug extremely complex systems when I don’t really need to. All those DIY alternatives are there not because people are struggling with NIH syndrome but because it is orders of magnitude simpler. YMMV. Or if you want to make K8s do something it isn’t designed or not presently capable of, so you need to hack on it. Grokking that codebase requires a lot of brainpower and time (I tried and I gave up, deciding I don’t want to, until the day finally comes and I must.)
  - Alupis 2y ago
    This is the other common pushback on using k8s, and it's usually unfounded. Very rarely will you ever need to troubleshoot k8s itself, if ever. I highly doubt most situations, especially ones like this article, will ever encounter an actual k8s issue, let alone discover a problem that requires digging into the k8s codebase. If you are doing complex stuff, then ya, it will be complex. If you are not doing complex stuff and find yourself thinking you need to troubleshoot k8s and/or dig into the codebase - then you are very clearly doing something wrong. A homebrewed alternative will also be complex in those cases, and most likely more complex than k8s because of all the non-standard duct tape you need apply.
    - drdaeman 2y ago
      Hah. Okay, here's an anecdote. First time I had to look under the hood (and discover it's not exactly easy to get through the wiring) is when I set up my first homelab cluster. I wanted to run it over an IPv6-only network I had, and it while the control plane worked data plane did not - turned out that some places were AF_INET only. That was quite a while ago, and it's all totally okay. But it was documented exactly nowhere except for some issue tracker, and it wasn't easy to get through CNI codebase to find enough information to find that issue. So, it wasn't doing something wrong, but I had to look into it. So I ran that cluster over IPv4. But it was a homelab, not some production-grade environment - so, commodity hardware, flaky networks and chaos monkeys. And I had it failing in various ways quite frequently. Problems of all kind, mostly network-related - some easy and well-documented like nf_conntrack capacity issues, some were just weird (spontaneous intermittent unidirectional connectivity losses that puzzled me, and the underlying network was totally okay, it was something in the CNI - I had to migrate off Weave to, IIRC, Calico, as Weave was nicer to configure but too buggy and harder to debug). I also ran a production GKE cluster, and - indeed - it failed much less often, but I still had to scrape and rebuild it once, because something had happened to it, and by then I knew that I'm simply not in capacity of proper diagnostics (but creating new nodes and discarding old ones, so all failing payloads migrate is easy - except, of course, that it solves nothing in the long term). In the end, for the homelab, I realized I don't need the flexibility of K8s. I briefly experimented with Nomad, and realized I don't even need Nomad for what I'm running. All I essentially wanted was failover, some replicated storage, and a private network underneath. And there are plenty of already robust age-proved tools for doing all of those. E.g. I needed a load balancer, but not specifically K8s load balancer. I know nginx and traefik a little bit, including some knowledge of of their codebase (I debugged both, successfully) so I just picked one. Same for the networking and storage, I just built what I needed from the well-known pieces with a bit of Nix glue, much more simpler and direct/hardcoded rather than managed by CNI stack. Most importantly, I didn't need fancy dynamic scaling capacities at all, as all my loads were and still are fairly static. As for that GKE cluster - it worked, until the company failed (for entirely non-technical reasons with upper management). If you know what I did wrong, in general - please tell. I'm not entirely dismissing K8s (it's nice to have its capabilities), but currently have a strong opinion that it's way too complex to be used when you don't have a team or person who can actually understand it. Just like you wouldn't run GNU/Linux or FreeBSD on server back in '00s without having an on-call sysadmin who knows how to deal with it (I was one, freezing my ass in the server room, debugging NIC driver crashing kernel at 2am - fun times (not really)!, and maybe I grew dumber but a lot of Linux source code is somewhat comprehensible to me, while K8s is much harder to get through.) This is all just my personal experience and my personal attitude to how I do things (I just hate not knowing how things in my care/responsibility work, because I feel helpless). As always, YMMV.
OJFord 2y ago
If you stood up k8s just for an IRC client that would definitely be silly, but honestly if you already have it (in this context presumably as the way you self-host stuff at home) then I don't really think there's anything silly about it. If anything it would be silly to have all your 'serious' stuff in k8s and then a little systemd unit on one of the nodes (or a separate instance/machine) or something for your IRC client! (Before I'm accused of cargo-culting: I don't run it at home, and just finished moving off it at work last week.)

Alupis 2y ago

People so very often are quick to look down upon anything using k8s these days, often citing complexity and some hand-wavy statement about "you're not web scale" or similar. These statements are usually accompanied by complex suggestions as "more simple" alternatives. On one hand you can manually provision a VM, configure the VM's firewall, update the OS, install several packages, write a bash script which can be used with cron, configure cron, setup backups, and more, followed by routine maintenance that is required to keep this system running and secure. This dance has to be done every time you deploy the app... oh, and don't forget that hack you did one late night to get your failing app back online but forgot to document! Or, on the other hand, you can write 1-3 relatively simple yaml files, just once, that explicitly describe what your app should look like within the cluster, and then it just does it's thing forever. k8s is very flexible and very useful, even for simple things. Why would you not use it, should be the default question.

drdaeman 2y ago
Setting up and maintaining along the happy path is not an issue, it’s well-documented and has existing automations if desirable so you can set up a cluster in just a few minutes (if you have the experience already). And YAML files are nearly trivial in most cases. Boilerplate is not pretty but it’s easy. The only Problem (with a capital P) with K8s is when it suddenly and unexpectedly breaks down - something fails and you need to figure out what is going on. That’s where its complexity bites you, hard - because it’s a whole new OS on top of another OS - so, a whole new giant behemoth to learn and debug. If you’re lucky it’s already documented by someone so you can just follow the recipe, but if luck runs out (it always eventually does) it’s not a common case you’re going to have some very unpleasant time. I’m a lazy ass, I hate having to debug extremely complex systems when I don’t really need to. All those DIY alternatives are there not because people are struggling with NIH syndrome but because it is orders of magnitude simpler. YMMV. Or if you want to make K8s do something it isn’t designed or not presently capable of, so you need to hack on it. Grokking that codebase requires a lot of brainpower and time (I tried and I gave up, deciding I don’t want to, until the day finally comes and I must.)
- Alupis 2y ago
  This is the other common pushback on using k8s, and it's usually unfounded. Very rarely will you ever need to troubleshoot k8s itself, if ever. I highly doubt most situations, especially ones like this article, will ever encounter an actual k8s issue, let alone discover a problem that requires digging into the k8s codebase. If you are doing complex stuff, then ya, it will be complex. If you are not doing complex stuff and find yourself thinking you need to troubleshoot k8s and/or dig into the codebase - then you are very clearly doing something wrong. A homebrewed alternative will also be complex in those cases, and most likely more complex than k8s because of all the non-standard duct tape you need apply.
  - drdaeman 2y ago
    Hah. Okay, here's an anecdote. First time I had to look under the hood (and discover it's not exactly easy to get through the wiring) is when I set up my first homelab cluster. I wanted to run it over an IPv6-only network I had, and it while the control plane worked data plane did not - turned out that some places were AF_INET only. That was quite a while ago, and it's all totally okay. But it was documented exactly nowhere except for some issue tracker, and it wasn't easy to get through CNI codebase to find enough information to find that issue. So, it wasn't doing something wrong, but I had to look into it. So I ran that cluster over IPv4. But it was a homelab, not some production-grade environment - so, commodity hardware, flaky networks and chaos monkeys. And I had it failing in various ways quite frequently. Problems of all kind, mostly network-related - some easy and well-documented like nf_conntrack capacity issues, some were just weird (spontaneous intermittent unidirectional connectivity losses that puzzled me, and the underlying network was totally okay, it was something in the CNI - I had to migrate off Weave to, IIRC, Calico, as Weave was nicer to configure but too buggy and harder to debug). I also ran a production GKE cluster, and - indeed - it failed much less often, but I still had to scrape and rebuild it once, because something had happened to it, and by then I knew that I'm simply not in capacity of proper diagnostics (but creating new nodes and discarding old ones, so all failing payloads migrate is easy - except, of course, that it solves nothing in the long term). In the end, for the homelab, I realized I don't need the flexibility of K8s. I briefly experimented with Nomad, and realized I don't even need Nomad for what I'm running. All I essentially wanted was failover, some replicated storage, and a private network underneath. And there are plenty of already robust age-proved tools for doing all of those. E.g. I needed a load balancer, but not specifically K8s load balancer. I know nginx and traefik a little bit, including some knowledge of of their codebase (I debugged both, successfully) so I just picked one. Same for the networking and storage, I just built what I needed from the well-known pieces with a bit of Nix glue, much more simpler and direct/hardcoded rather than managed by CNI stack. Most importantly, I didn't need fancy dynamic scaling capacities at all, as all my loads were and still are fairly static. As for that GKE cluster - it worked, until the company failed (for entirely non-technical reasons with upper management). If you know what I did wrong, in general - please tell. I'm not entirely dismissing K8s (it's nice to have its capabilities), but currently have a strong opinion that it's way too complex to be used when you don't have a team or person who can actually understand it. Just like you wouldn't run GNU/Linux or FreeBSD on server back in '00s without having an on-call sysadmin who knows how to deal with it (I was one, freezing my ass in the server room, debugging NIC driver crashing kernel at 2am - fun times (not really)!, and maybe I grew dumber but a lot of Linux source code is somewhat comprehensible to me, while K8s is much harder to get through.) This is all just my personal experience and my personal attitude to how I do things (I just hate not knowing how things in my care/responsibility work, because I feel helpless). As always, YMMV.

OJFord 2y ago

If you stood up k8s just for an IRC client that would definitely be silly, but honestly if you already have it (in this context presumably as the way you self-host stuff at home) then I don't really think there's anything silly about it. If anything it would be silly to have all your 'serious' stuff in k8s and then a little systemd unit on one of the nodes (or a separate instance/machine) or something for your IRC client! (Before I'm accused of cargo-culting: I don't run it at home, and just finished moving off it at work last week.)