5 ms·

My opinion at the moment, subject to change: Docker and Kubernetes have a high price as they add a lot of complexity and make debugging harder. Especially for n

My opinion at the moment, subject to change: Docker and Kubernetes have a high price as they add a lot of complexity and make debugging harder. Especially for networking and state-full. Most of the problems they claim to solve where already solved years ago by more mature solutions: - reproducible builds and dependencies management: distribution package managers - high availability: load balancers, keepalive, dynamic routing, clustering and most importantly: data replication (where is this in k8s?) since downtime is nothing compared to data loss - resource limitation: cgroups - isolation: the good old user and groups, vlans, firewalls - orchestration, rollouts, rollbacks: CD pipelines, configuration management Technically the only thing I find easier with k8s is bin packing. The real advantage I find in containers and k8s is they became a standard. You don't have to learn the architecture that each company has designed for itself and dig into custom built solutions. So you can easily switch jobs and they can easily replace you. If you throw that away with custom container implementations for each PaaS offering and fragmentation of orchestrator solutions then you only get the complexity and no benefit. We're going to see in a few more years where it goes but I'm not exactly optimistic.

dilyevsky 5y ago
“Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”
- dijit 5y ago
  There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.
  - dilyevsky 5y ago
    It maybe a fallacy alright but this exactly what usually happens
- stephenr 5y ago
  TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.
  - robertlagrant 5y ago
    Be cool to hear some examples/war stories. Do you have a blog?
    - bsaul 5y ago
      same here. Really interested to read about the popular ways of screwing a k8s cluster
      - stephenr 5y ago
        I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
        robertlagrant 5y ago
        Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
        stephenr 5y ago
        Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
- trabant00 5y ago
  What a low effort and low value post. Where exactly in my list of alternative solutions did you find bash?
  - dilyevsky 5y ago
    There’s like 10 different things in your post, how exactly you suggest we automate all that stuff? Usually that ends up being bash possibly with some hcl/cf/ansible mixed in (not a whole lot better). If you suggest to write python or go to do it that’s imo even worse as you then end up with bespoke version of <50% of nomad/k8s that only you understand.
    - trabant00 5y ago
      Why would it be "bash possibly with some hcl/cf/ansible mixed in"? There's provisioning to be done using terraform or pxe installer, packages to be installed, config files to be copied, unit files to be edited, services to be started. That's 100% configuration management, no need for bash, python or go. Although you will get with any solution at a point where text configuration is too large and you get into templating and then coding, and you will get there with K8s as well - see pulumi, dhall, etc There's people who can handle complexity with an elegant solutions and people who can't and search for silver bullets like K8s.
throwaway_4ever 5y ago
> - reproducible builds and dependencies management: distribution package managers As in? What distro package manager is allowing me to give a project newbie a single command to hit the ground running across all platforms, "docker-compose up"? What distro package manager is allowing me to have several different platforms that can just grab a GitHub repo link and deploy it viola?
- trabant00 5y ago
  I don't understand the "on all platforms" requirement. Why do you need that? Why not choose one and have consistent dev, test and prod envs?
  - throwaway_4ever 5y ago
    The target platform is the same, but the dev environments are different. People will prefer their own MacOS, Windows, or Linux based setups, and all the different headaches of various conflicting libraries and architectures (ARM v. x86). Docker seems much simpler than having them setup a matching VM or cloud desktop. The different platforms for deploying is about cloud-agnosticism, avoiding vendor lock-in by using an easily reproducible standard. Even for just one cloud deployment, Ansible and similar tools aren't as simple and reproducible as a docker-compose.yml and Dockerfile. I'm not saying I've started from 0 and tried all the comparable solutions here, comparing and contrasting them. It's that when I looked at the lay of the land, asked "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest.
    - trabant00 5y ago
      > People will prefer their own MacOS, Windows, or Linux based setups As a devops I strongly oppose developing on the desktop OS. Only your repos and code editor should be on your workstation, and you should have dedicated dev environment in a server farm - be that vms or containers. > "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest. Only you don't deploy Docker in prod, do you? The fact that the container starts means nothing if it doesn't interface with the other contanerized apps. So you start needing a Kubernetes cluster locally and all goes to hell from there.

dilyevsky 5y ago

“Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”

dijit 5y ago
There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.
- dilyevsky 5y ago
  It maybe a fallacy alright but this exactly what usually happens
stephenr 5y ago
TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.
- robertlagrant 5y ago
  Be cool to hear some examples/war stories. Do you have a blog?
  - bsaul 5y ago
    same here. Really interested to read about the popular ways of screwing a k8s cluster
    - stephenr 5y ago
      I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
      - robertlagrant 5y ago
        Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
        stephenr 5y ago
        Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
trabant00 5y ago
What a low effort and low value post. Where exactly in my list of alternative solutions did you find bash?
- dilyevsky 5y ago
  There’s like 10 different things in your post, how exactly you suggest we automate all that stuff? Usually that ends up being bash possibly with some hcl/cf/ansible mixed in (not a whole lot better). If you suggest to write python or go to do it that’s imo even worse as you then end up with bespoke version of <50% of nomad/k8s that only you understand.
  - trabant00 5y ago
    Why would it be "bash possibly with some hcl/cf/ansible mixed in"? There's provisioning to be done using terraform or pxe installer, packages to be installed, config files to be copied, unit files to be edited, services to be started. That's 100% configuration management, no need for bash, python or go. Although you will get with any solution at a point where text configuration is too large and you get into templating and then coding, and you will get there with K8s as well - see pulumi, dhall, etc There's people who can handle complexity with an elegant solutions and people who can't and search for silver bullets like K8s.

throwaway_4ever 5y ago

> - reproducible builds and dependencies management: distribution package managers As in? What distro package manager is allowing me to give a project newbie a single command to hit the ground running across all platforms, "docker-compose up"? What distro package manager is allowing me to have several different platforms that can just grab a GitHub repo link and deploy it viola?

trabant00 5y ago
I don't understand the "on all platforms" requirement. Why do you need that? Why not choose one and have consistent dev, test and prod envs?
- throwaway_4ever 5y ago
  The target platform is the same, but the dev environments are different. People will prefer their own MacOS, Windows, or Linux based setups, and all the different headaches of various conflicting libraries and architectures (ARM v. x86). Docker seems much simpler than having them setup a matching VM or cloud desktop. The different platforms for deploying is about cloud-agnosticism, avoiding vendor lock-in by using an easily reproducible standard. Even for just one cloud deployment, Ansible and similar tools aren't as simple and reproducible as a docker-compose.yml and Dockerfile. I'm not saying I've started from 0 and tried all the comparable solutions here, comparing and contrasting them. It's that when I looked at the lay of the land, asked "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest.
  - trabant00 5y ago
    > People will prefer their own MacOS, Windows, or Linux based setups As a devops I strongly oppose developing on the desktop OS. Only your repos and code editor should be on your workstation, and you should have dedicated dev environment in a server farm - be that vms or containers. > "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest. Only you don't deploy Docker in prod, do you? The fact that the container starts means nothing if it doesn't interface with the other contanerized apps. So you start needing a Kubernetes cluster locally and all goes to hell from there.