4 ms·

“Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”

by dilyevsky 5y ago

dijit 5y ago
There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.
- dilyevsky 5y ago
  It maybe a fallacy alright but this exactly what usually happens
stephenr 5y ago
TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.
- robertlagrant 5y ago
  Be cool to hear some examples/war stories. Do you have a blog?
  - bsaul 5y ago
    same here. Really interested to read about the popular ways of screwing a k8s cluster
    - stephenr 5y ago
      I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
      - robertlagrant 5y ago
        Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
        stephenr 5y ago
        Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
        robertlagrant 5y ago
        We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
        stephenr 5y ago
        My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
        robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
    - dijit 5y ago
      You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ
  - stephenr 5y ago
    > Do you have a blog No, I have a job. Seriously though, no. As a one-man band, anything besides "client X needs Y" or "[internal/forked/contributed to] tool A needs enhancement/bugfix B" tends to be very low priority.
- KronisLV 5y ago
  > TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. While that's a fair position to take, you also might be in the minority here. I've seen plenty of in-house solutions in corporate environments and one of the things that they've all had in common was how bad they were, due to them being a way to just get things done and no more (e.g. no proper testing, no proper documentation, bad/wrong abstractions). It's not like your (or anyone else's) collection of Bash scripts can compete with open source solutions that have had thousands of collective man years put into making them better and addressing the never ending churn of keeping things working across new distro releases etc. I'm not advocating to go straight to Kubernetes - Docker Swarm or Hashicorp Nomad, or even just Docker Compose can all be decent alternatives that are simpler to wrap one's head around and manage, in most smaller deployments. As for no one knowing what containers do, i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat, with who knows what sort of Cthulhu living in the shared libraries folder and everything being so brittle that you cannot even look at it the wrong way, or it'll break and you'll spend days digging through obscure error logs. Worse yet, have you ever needed to manage an environment that has applications that need 4 different JDK versions and many different launch methods (some use systemd services with Tomcat, some use Tomcat launch scripts, some have just a .jar that needs to be launched etc.), yet still count upon some shared environment configuration being present? Give me individual containers over that mess any day. I think that there should be a clear separation between what a server needs to keep operating and where the "business software" would run - basically an /app directory with a subdirectory per application with whatever it needs, be it bind mounted data storage, configuration files, log output (in lieu of log shipping) etc., as opposed to having some stuff be strewn across /var, /etc, /opt, /www and who knows how many other directories. Containers give me just that.
  - stephenr 5y ago
    > you also might be in the minority here I'd be astounded if I'm not. The tech industry is ridiculously cargo-culty. I never said I expect the majority of people to agree with me, I said what I prefer. The majority of people probably think McDonalds is just fine, and that Google is "not so bad" too, and it doesn't bother me that other people disagree with my opinion. If you ask me to fix your fucked infra because someone threw together a Rube Goldberg like Kubernetes environment just to achieve rolling deployments of their application, I'm going to tell you it's a shit solution, just as if you asked me to critique a McDonalds hamburger I'm going to tell you it's a shit hamburger. Me saying it's shit doesn't mean you can't use the thing, it means my opinion is that there are better ways to solve the problem/achieve the desired outcome, and you can either accept that opinion and make a change, or disagree and I'll go provide my services to another client. It's really that simple. > thousands of collective man years put into making them better They've had a ridiculous large number of man years working on them, no question. The whole point is that we disagree about whether they're "better" or not. Adding complexity is not necessarily making something better. > i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat In... a decade and a half of doing this I think I've been asked to support one JVM based thing, and that was quite a while ago. I'm sure it is a shit show, but I'm also sure that there are better alternatives than 8 more layers of abstraction. Lightweight VM-like containers are literally a thing that exists, and can give essentially the same provisioning as a traditional VM, with the "lightweight" / "shared resources" aspect of a container solution.
trabant00 5y ago
What a low effort and low value post. Where exactly in my list of alternative solutions did you find bash?
- dilyevsky 5y ago
  There’s like 10 different things in your post, how exactly you suggest we automate all that stuff? Usually that ends up being bash possibly with some hcl/cf/ansible mixed in (not a whole lot better). If you suggest to write python or go to do it that’s imo even worse as you then end up with bespoke version of <50% of nomad/k8s that only you understand.
  - trabant00 5y ago
    Why would it be "bash possibly with some hcl/cf/ansible mixed in"? There's provisioning to be done using terraform or pxe installer, packages to be installed, config files to be copied, unit files to be edited, services to be started. That's 100% configuration management, no need for bash, python or go. Although you will get with any solution at a point where text configuration is too large and you get into templating and then coding, and you will get there with K8s as well - see pulumi, dhall, etc There's people who can handle complexity with an elegant solutions and people who can't and search for silver bullets like K8s.

“Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”

dijit 5y ago

There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.

dilyevsky 5y ago
It maybe a fallacy alright but this exactly what usually happens

stephenr 5y ago

TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.

robertlagrant 5y ago
Be cool to hear some examples/war stories. Do you have a blog?
- bsaul 5y ago
  same here. Really interested to read about the popular ways of screwing a k8s cluster
  - stephenr 5y ago
    I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
    - robertlagrant 5y ago
      Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
      - stephenr 5y ago
        Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
        robertlagrant 5y ago
        We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
        stephenr 5y ago
        My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
        robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
  - dijit 5y ago
    You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ
- stephenr 5y ago
  > Do you have a blog No, I have a job. Seriously though, no. As a one-man band, anything besides "client X needs Y" or "[internal/forked/contributed to] tool A needs enhancement/bugfix B" tends to be very low priority.
KronisLV 5y ago
> TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. While that's a fair position to take, you also might be in the minority here. I've seen plenty of in-house solutions in corporate environments and one of the things that they've all had in common was how bad they were, due to them being a way to just get things done and no more (e.g. no proper testing, no proper documentation, bad/wrong abstractions). It's not like your (or anyone else's) collection of Bash scripts can compete with open source solutions that have had thousands of collective man years put into making them better and addressing the never ending churn of keeping things working across new distro releases etc. I'm not advocating to go straight to Kubernetes - Docker Swarm or Hashicorp Nomad, or even just Docker Compose can all be decent alternatives that are simpler to wrap one's head around and manage, in most smaller deployments. As for no one knowing what containers do, i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat, with who knows what sort of Cthulhu living in the shared libraries folder and everything being so brittle that you cannot even look at it the wrong way, or it'll break and you'll spend days digging through obscure error logs. Worse yet, have you ever needed to manage an environment that has applications that need 4 different JDK versions and many different launch methods (some use systemd services with Tomcat, some use Tomcat launch scripts, some have just a .jar that needs to be launched etc.), yet still count upon some shared environment configuration being present? Give me individual containers over that mess any day. I think that there should be a clear separation between what a server needs to keep operating and where the "business software" would run - basically an /app directory with a subdirectory per application with whatever it needs, be it bind mounted data storage, configuration files, log output (in lieu of log shipping) etc., as opposed to having some stuff be strewn across /var, /etc, /opt, /www and who knows how many other directories. Containers give me just that.
- stephenr 5y ago
  > you also might be in the minority here I'd be astounded if I'm not. The tech industry is ridiculously cargo-culty. I never said I expect the majority of people to agree with me, I said what I prefer. The majority of people probably think McDonalds is just fine, and that Google is "not so bad" too, and it doesn't bother me that other people disagree with my opinion. If you ask me to fix your fucked infra because someone threw together a Rube Goldberg like Kubernetes environment just to achieve rolling deployments of their application, I'm going to tell you it's a shit solution, just as if you asked me to critique a McDonalds hamburger I'm going to tell you it's a shit hamburger. Me saying it's shit doesn't mean you can't use the thing, it means my opinion is that there are better ways to solve the problem/achieve the desired outcome, and you can either accept that opinion and make a change, or disagree and I'll go provide my services to another client. It's really that simple. > thousands of collective man years put into making them better They've had a ridiculous large number of man years working on them, no question. The whole point is that we disagree about whether they're "better" or not. Adding complexity is not necessarily making something better. > i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat In... a decade and a half of doing this I think I've been asked to support one JVM based thing, and that was quite a while ago. I'm sure it is a shit show, but I'm also sure that there are better alternatives than 8 more layers of abstraction. Lightweight VM-like containers are literally a thing that exists, and can give essentially the same provisioning as a traditional VM, with the "lightweight" / "shared resources" aspect of a container solution.

trabant00 5y ago

What a low effort and low value post. Where exactly in my list of alternative solutions did you find bash?

dilyevsky 5y ago
There’s like 10 different things in your post, how exactly you suggest we automate all that stuff? Usually that ends up being bash possibly with some hcl/cf/ansible mixed in (not a whole lot better). If you suggest to write python or go to do it that’s imo even worse as you then end up with bespoke version of <50% of nomad/k8s that only you understand.
- trabant00 5y ago
  Why would it be "bash possibly with some hcl/cf/ansible mixed in"? There's provisioning to be done using terraform or pxe installer, packages to be installed, config files to be copied, unit files to be edited, services to be started. That's 100% configuration management, no need for bash, python or go. Although you will get with any solution at a point where text configuration is too large and you get into templating and then coding, and you will get there with K8s as well - see pulumi, dhall, etc There's people who can handle complexity with an elegant solutions and people who can't and search for silver bullets like K8s.