12 ms·

> “The biggest mistake was to miss Kubernetes. We were in that collective thought bubble where internally we thought Kubernetes was way too complicated and Swar

by zzyzxd 5y ago

> “The biggest mistake was to miss Kubernetes. We were in that collective thought bubble where internally we thought Kubernetes was way too complicated and Swarm would be much more successful,” Jérôme Petazzoni, one of Docker’s first and longest serving employees, said. “It was our collective failure to not realize that.” They were not wrong on saying that Kubernetes was very complicated, at least in some sense. In the beginning no one wanted to use it because they could easily setup Docker Swarm with minimal effort. This argument still pops up frequently on HN when there's a new post about Kubernetes. I guess the problem was they didn't realize why Kubernetes needs to be that complicated. And if a system is complicated for good reasons, that's actually good business opportunity and tons of people and companies will be willing to make the effort to fill that gap.

gizdan 5y ago
On this note, HN users and several HC employees (including some enterprise architects) I've spoken to love to mention Nomad and how it's much lighter and simpler to run, completely oblivious that many with bigger names have tried and failed to go up against K8s. Kubernetes has become a speeding bullet, it won't be easy to catch and it has to run through a lot of walls to slow down for a competitor to do any damage to it's dominance.
- schmichael 5y ago
  > completely oblivious that many with bigger names have tried and failed to go up against K8s Ha, I can at least assure you we're not oblivious. Nomad has a product manager from Mesosphere and most of HashiCorp's products integrate deeply with Kubernetes. We're well aware Kubernetes is the juggernaut to which every other scheduler has succumbed. I believe there's room for both Nomad and Kubernetes. Whether as competitors or complements, having more than one vibrant project in the orchestration space will hopefully make all the projects better for users. Any one project has to make tradeoffs that improve the user experience for some while degrading it for others. For example Kubernetes has an IP-per-Pod network abstraction which provides an excellent out of the box PaaS-like experience for microservices in the cloud. On the other hand Nomad's more flexible network model more easily supports networks outside of flat cloud topologies whether it's small home raspberrypi clusters or globally distributed edge compute.
  - gizdan 5y ago
    Sorry, I think I came across dismissive of Nomad which I am not. I work with multiple products in the HC stack daily, but have yet to work with Nomad, however, the bits that I have seen and read about Nomad, it seems like a great tool, and I agree there is place for Nomad in the current ecosystem, just like there is place for ECS. Even if I didn't think so, at the very least, like you said, it's good to have competition. What I was calling out is the abundance of people who claim it is the greatest thing since the tech equivalent of sliced bread, and love to point out how simple it is despite requiring the need for Consul to run it at enterprise scale and likely also Vault if you need secret management in any form, which then also needs Consul if you want to run it at scale. I have intimate experience with running Vault and Consul and have advised colleagues for the better part of 2 years on utilising Vault (either as basic secrets management or some of its more advanced features). IIRC the recommendation from HC is also running a third Consul cluster for service discovery. If running Nomad at scale is anything like Vault, then it isn't as simple as people make it out to be, never mind the fact that you'll probably be running 5 different clusters of 3 different products to provide functionality that doesn't fulfill half of what Kubernetes gives.
    - baranoff 5y ago
      Great perspective. Thank you for you comment!
    - schmichael 5y ago
      > If running Nomad at scale is anything like Vault, then it isn't as simple as people make it out to be, never mind the fact that you'll probably be running 5 different clusters of 3 different products to provide functionality that doesn't fulfill half of what Kubernetes gives. The complexity of setting up a Nomad cluster due to Vault and Consul being soft requirements is very real and something we're hoping to (finally!) make progress on in the coming months.
      - merb 5y ago
        you would need to make it as simple as running k3s on 1,3,5 nodes and probably also have stuff like system-upgrade-manager and another good idea is to work on flatcar linux, etc. k3s is so good for small-middle sized metal clusters and also works on a single node. it's so simple to start, you will have a rough time to actually try to gain momentum in this space which will also not generate a lot of revenue. the only thing which is not so easy is LoadBalancing to extern i.e. kube-vip/metallb both can be a pain (externalTrafficPolicy: Local is still a shit show). k3s basically creates a k8s cluster with everything except good load balancing over multiple nodes
- lenkite 5y ago
  We will see if the next hot scaling/clustering/orchestration solution replaces k8s in a decade from now. Fingers crossed. There have been lots of speeding, hard-to-catch bullets in the past.
- scaryclam 5y ago
  Thing is, Nomad isn't trying to go up against K8S. It's part of the Hashicorp family of products, so if you use a couple of those, it's easy to add Nomad into the mix, or just use K8S. While other competitors have tried to out-do K8S, Hashicorp seem to try to work with the broader eco-system, which means some will just use Nomad to keep it simple, but don't need to for Hashicorp to continue working for them in other areas.
  - gizdan 5y ago
    > Thing is, Nomad isn't trying to go up against K8S. Hard[0] disagree[1] on this[2]. Nomad is a direct competitor of Kubernetes. [0] https://www.nomadproject.io/docs/nomad-vs-kubernetes https://www.nomadproject.io/docs/nomad-vs-kubernetes [1] https://www.hashicorp.com/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad https://www.hashicorp.com/blog/a-kubernetes-user-s-guide-to-... [2] https://www.hashicorp.com/blog/nomad-kubernetes-a-pragmatic-look-at-choosing-orchestrators https://www.hashicorp.com/blog/nomad-kubernetes-a-pragmatic-...
    - jokethrowaway 5y ago
      Nomad can compete with a fraction of k8s, a bit like all the other hashicorp products
throwaway20371 5y ago
The ironic thing is that Docker became successful only because of the tremendous hype they generated about themselves, and then failed to see the K8s hype train barreling towards them. Swarm is still what 99% of companies need when they adopt Kubernetes, and is probably 1/10th to 1/20th the cost. Somebody needs to re-invent swarm as an independent project so that we can stop investing in K8s. The cost of K8s isn't even in the complexity, it's in all the millions of side-effects of the complexity, like security, fragmentation, and specialization.
- cpuguy83 5y ago
  Swarm is (and always has been) an independent project: github.com/docker/swarmkit
waspight 5y ago
What is so hard about k8s? I think for example aws with fargate and all the networking there is far more complicated… i have never used swarm though.
- angelzen 5y ago
  k8s is just a bunch of processes working together. The downside is that if something goes wrong, it is unclear which process is faulty and how to fix it. A misconfigured DNS may take down the whole cluster, and the symptoms are that every process fails because the network is out. Difficult to backtrack to the fault source, there are a lot of configs pertaining to the network that could be faulty. On the plus side, if someone else configured the cluster correctly for you (e.g. a Cloud service like GKE), then it's a breeze to use.
- geocar 5y ago
  If you're talking about a hosted k8s like EKS or a toy/single-node k8s, in 2021, then nothing: k8s is much better. But if you're on-prem, and have a tonne of metal and just arrived in 2014 via a time machine, swarm was so much simpler and simpler that if you were a sysadmin who already had their own scripts -- their own jenkins-powered IC and git-hooks that built and deployed -- whatever it is you were building -- then swarm looked like a nice gradual extension of that, and k8s looked more like starting over and admitting defeat.
  - waspight 5y ago
    I see your point there. I only work on the pod side, not touching any metal.
    - runlevel1 5y ago
      There's still plenty of room for improvement, but managing a K8s cluster on-prem is miles better than it was in 2014.
  - Spivak 5y ago
    I mean k8s is basically an infra rewrite in any shop that was/is currently VM based. "Hey you know that fundamental unit of isolation you've built all your tooling around. Throw it all away." Don't get me wrong, I would absolutely go with k8s on any greenfield project but there's a huge huge opportunity for someone to take everyone's VM orchestration tools and quietly and semi-transparently add k8s support for gradual migrations.
rualca 5y ago
> I guess the problem was they didn't realize why Kubernetes needs to be that complicated. I guess this strongly depends on one's needs. I mean, if all you want to do is glue together a couple of nodes and have a few stacks running on them, do we really need all the bells and whistles? I mean, with Docker Swarm anyone can get ingress right from the start on any bare ones cloud provider. Deploy your Docker Swarm cluster, slap a service like Traefik, and you're done. It's a 10min job. What else does Kubernetes brings to the table?
sgk284 5y ago
K8S is great if you have Google problems, but most people don't and I think much of hype around it is much like the hype that existed a decade ago around "web scale" and "big data" and "NoSQL for everything". Docker Swarm is (with some use-case specific exceptions) more than sufficient for any shop running a few hundred nodes with a few thousand containers – and it is two orders of magnitude less complex than k8s. You can read Docker Swarm's docs end-to-end in < 30 minutes, and have a cluster running in roughly the amount of time it takes to install Docker on three boxes. Docker's failing was a failure to market Docker Swarm. Most people don't even know it exists, yet every single machine with Docker has it already! The last time I checked, they only had a single dev that was part-time allocated to supporting it. So it's not so much that Docker failed to jump on the k8s bandwagon, so much as they built a wonderful thing that they never marketed and semi-abandoned, to focus on Docker Hub and other core Docker things they thought they'd be able to more directly monetize. Fortunately, Docker Swarm remains a wonderful piece of software, even without particularly active investment, because it is a fairly complete offering for most use cases. And it seems to be growing in adoption.
- bovermyer 5y ago
  I was dismayed when Docker effectively (and quietly) declared that they were no longer supporting Docker Swarm. Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes. I just can't choose it over K8s, though, knowing that it's unsupported.
  - sgk284 5y ago
    I've hemmed and hawed over this as well, but I believe that sometimes software can be complete and stable and not need constant iteration. As long as they add security patches and don't remove it from Docker, I'll still default to Swarm. My fallback, if Swarm is ever removed, is Hashicorp's Nomad. I've only used it in a hobby capacity, but is (at face value, at least) another wonderful piece of software. I've used k8s a bunch in production (including right now at my current employer) and just can't recommend it – it's a technical distraction for most teams.
technicolorwhat 5y ago
I think docker itself had plenty of time to innovate. They could have solved the problem long before mainstream got a hold of kubernetes. They just didn't solve the actual problems users where having at that time, which was container orchestration across multiple servers. They went very very deep with a lot of drivers and API changes and what not. If I am not mistaken docker-compose wasn't even a part of docker first before it got bundled, someone else was solving docker's same-server orchestration problems. Kubernetes is for a lot of organisations pretty difficult and way too much for a start. We used rancher for solving our problems which did a fine job, rancher got a worse when they made the move to Kubernetes. In the beginning they (docker) also removed a lot of options like cpu throttling and configuration and such that LXC had from the start. They also had their own version (a shim) of pid 1, some point at the time. And other things that made it a little painful to properly containerise. I was often very frustrated by docker and fell back on lxc. There was also something with the management of docker images like pruning and cleaning and other of much needed functionality that just didn't get included. Also something with the container registries which I a this point can't remember (maybe deleting images or authentication out of the box or something that made it hard to host yourself). Anyway I think it's failed because it failed to listen to its users and act upon them. They really had a lot of chances I think. I think they just made some wrong business decisions. I always felt they had a strong technical CTO that really was deep on the product but not on the whole pipeline from dev->x->prod workflows.
- V99 5y ago
  > We used rancher for solving our problems which did a fine job, rancher got a worse when they made the move to Kubernetes. (Early Rancher employee) We liked our Cattle orchestration and ease of use of 1.x as much as the next person. Hell, I still like a lot of it better. But just as this article talks about with Swarm, embracing K8s was absolutely the right move. We were the smallest of at least 4 major choices. Picking the right horse early enough and making K8s easier to use led us to a successful exit and continued relevance (now in SUSE) instead of a slow painful spiral to irrelevance like Swarm, Mesos, and others and eventual fire-sale.
  - 5y ago
erulabs 5y ago
It's an incredible litmus test for developer v sysadmin. Developers think the complexity is unnecessary and so don't want to learn k8s. For sysadmins it's mostly just nice automation around all the chores we used to do with home-brewed bash scripts.
- ffdf3r3rf 5y ago
  As a developer, I hate k8s, but mostly because my management does not understand difference between developers and devops. They changed our job titles to devops engineers and fired sys admins. I understand k8s is complex for people who specialize in it, try imagining as a developer, you need to setup private vlan ingress. At least, on regular Linux, I know how to setup ip tables or look up 1000s of tutorial. K8s and devops has destroyed us developers' productivity, confidence, and morale.
  - sho 5y ago
    Isn't it your management who have destroyed those things?
    - ffdf3r3rf 5y ago
      I don't disagree but I don't know where they are getting their marketing material where they think developers can do devops. I suspect it might some sales tactics from these Cloud/DevOps companies.
- goodpoint 5y ago
  I've never met a sysadmin that would prefer it over the bash scripts.
  - 0des 5y ago
    What's your sample size?
    - goodpoint 5y ago
dilyevsky 5y ago
What I haven’t seen mentioned here is at least by my recollection swarm was incredibly bug ridden especially in networking layer around ‘17/18 when k8s really started pulling away
makeitdouble 5y ago
I’ve seen a number of companies going for k8s and not other solution not because they needed the complexity, but because it was first party supported. To me what killed Swarm is Google needing to spread k8s. If you are eyeing at GCE but decide you want a bit more flexibility, going for a Google backed solution gives you k8s, and Swarm isn’t even on the choice list. Choosing a third party means you’re on your own to bootstrap/make it live, or contract a vendor that might not live longer than you. That’s fine for companies who strategically choose to invest for the long term, sadly that’s not most of the companies out there.
wpietri 5y ago
> I guess the problem was they didn't realize why Kubernetes needs to be that complicated. Does it? It seems like there are three kinds of complexity involved here: the essential complexity of the total problem set, the essential complexity of a specific company's problems, and the amount of complexity experienced by a user as they try to solve those problems. Let's stipulate that Kubernetes needs to be that complicated for Google's needs. I'm not totally sure that's true, but let's pretend for now. So we'll say you're right for the first kind. But I'm still not seeing it for the other two. For modest deployment efforts, Kubernetes is a giant pain in the ass. It's just way more complicated than needed for many non-Google use cases. I tried setting up Kubernetes for a bunch of personal stuff I run and it was egregious overkill. I persisted until my home lighting system went dark because some internal Kubernetes keys expired and it decided that just not working was a good failure mode. Last time I mentioned this on HN several people said, well duh, obviously you shouldn't run it just for yourself. And that's fine; every technology has an expected operational envelope. But there are an awful lot of projects out there that have between 0.0 and 0.5 sysadmins. For them it's too complex. Kubernetes is just too complex for many people's needs. And the other side of that is complexity of experience. Modern processors are fantastically complicated. I'm on a mailing list for people in finance who micro-optimize everything to maximize response speed. Those people know incredible amounts about exactly what different processors do. And they have to. But most working developers know about 1% of that because a lot of people work very hard to hide the complexity from them. I don't think that really happened with Kubernetes. At this point, it is what it is. It's much easier to scale a technology up than down, so I expect it'll always be like this. But For whatever comes after Kubernetes, I'd like to see much more attention paid to more common use cases.
coding123 5y ago
It's really unfortunate. I am a kubernetes user today, but I LOVED swarm. K8S' main differentiator is that specific intentional "cross-cloud" features such as ingress and volume mapping that help the implementing cloud understand the deployment topology. It's basically the new PaaS but instead of working on one cloud it can work on any. I don't know that if Swarm had those features if it would really catch on like Kubernetes did. They didn't have the same budget as Google on that side of things.
SergeAx 5y ago
I beleive that wide k8s adoption is the same dead end as JS frameworks. It is not actually needed and is too complex, so it introduced it's own class of pain. VMs were a great abstraction with very little added complexity, so was Docker. K8s doesn't abstract anything away compared with Docker swarm, just adding unnecessary complexity.

by zzyzxd 5y ago

gizdan 5y ago

On this note, HN users and several HC employees (including some enterprise architects) I've spoken to love to mention Nomad and how it's much lighter and simpler to run, completely oblivious that many with bigger names have tried and failed to go up against K8s. Kubernetes has become a speeding bullet, it won't be easy to catch and it has to run through a lot of walls to slow down for a competitor to do any damage to it's dominance.

schmichael 5y ago
> completely oblivious that many with bigger names have tried and failed to go up against K8s Ha, I can at least assure you we're not oblivious. Nomad has a product manager from Mesosphere and most of HashiCorp's products integrate deeply with Kubernetes. We're well aware Kubernetes is the juggernaut to which every other scheduler has succumbed. I believe there's room for both Nomad and Kubernetes. Whether as competitors or complements, having more than one vibrant project in the orchestration space will hopefully make all the projects better for users. Any one project has to make tradeoffs that improve the user experience for some while degrading it for others. For example Kubernetes has an IP-per-Pod network abstraction which provides an excellent out of the box PaaS-like experience for microservices in the cloud. On the other hand Nomad's more flexible network model more easily supports networks outside of flat cloud topologies whether it's small home raspberrypi clusters or globally distributed edge compute.
- gizdan 5y ago
  Sorry, I think I came across dismissive of Nomad which I am not. I work with multiple products in the HC stack daily, but have yet to work with Nomad, however, the bits that I have seen and read about Nomad, it seems like a great tool, and I agree there is place for Nomad in the current ecosystem, just like there is place for ECS. Even if I didn't think so, at the very least, like you said, it's good to have competition. What I was calling out is the abundance of people who claim it is the greatest thing since the tech equivalent of sliced bread, and love to point out how simple it is despite requiring the need for Consul to run it at enterprise scale and likely also Vault if you need secret management in any form, which then also needs Consul if you want to run it at scale. I have intimate experience with running Vault and Consul and have advised colleagues for the better part of 2 years on utilising Vault (either as basic secrets management or some of its more advanced features). IIRC the recommendation from HC is also running a third Consul cluster for service discovery. If running Nomad at scale is anything like Vault, then it isn't as simple as people make it out to be, never mind the fact that you'll probably be running 5 different clusters of 3 different products to provide functionality that doesn't fulfill half of what Kubernetes gives.
  - baranoff 5y ago
    Great perspective. Thank you for you comment!
  - schmichael 5y ago
    > If running Nomad at scale is anything like Vault, then it isn't as simple as people make it out to be, never mind the fact that you'll probably be running 5 different clusters of 3 different products to provide functionality that doesn't fulfill half of what Kubernetes gives. The complexity of setting up a Nomad cluster due to Vault and Consul being soft requirements is very real and something we're hoping to (finally!) make progress on in the coming months.
    - merb 5y ago
      you would need to make it as simple as running k3s on 1,3,5 nodes and probably also have stuff like system-upgrade-manager and another good idea is to work on flatcar linux, etc. k3s is so good for small-middle sized metal clusters and also works on a single node. it's so simple to start, you will have a rough time to actually try to gain momentum in this space which will also not generate a lot of revenue. the only thing which is not so easy is LoadBalancing to extern i.e. kube-vip/metallb both can be a pain (externalTrafficPolicy: Local is still a shit show). k3s basically creates a k8s cluster with everything except good load balancing over multiple nodes
lenkite 5y ago
We will see if the next hot scaling/clustering/orchestration solution replaces k8s in a decade from now. Fingers crossed. There have been lots of speeding, hard-to-catch bullets in the past.
scaryclam 5y ago
Thing is, Nomad isn't trying to go up against K8S. It's part of the Hashicorp family of products, so if you use a couple of those, it's easy to add Nomad into the mix, or just use K8S. While other competitors have tried to out-do K8S, Hashicorp seem to try to work with the broader eco-system, which means some will just use Nomad to keep it simple, but don't need to for Hashicorp to continue working for them in other areas.
- gizdan 5y ago
  > Thing is, Nomad isn't trying to go up against K8S. Hard[0] disagree[1] on this[2]. Nomad is a direct competitor of Kubernetes. [0] https://www.nomadproject.io/docs/nomad-vs-kubernetes https://www.nomadproject.io/docs/nomad-vs-kubernetes [1] https://www.hashicorp.com/blog/a-kubernetes-user-s-guide-to-hashicorp-nomad https://www.hashicorp.com/blog/a-kubernetes-user-s-guide-to-... [2] https://www.hashicorp.com/blog/nomad-kubernetes-a-pragmatic-look-at-choosing-orchestrators https://www.hashicorp.com/blog/nomad-kubernetes-a-pragmatic-...
  - jokethrowaway 5y ago
    Nomad can compete with a fraction of k8s, a bit like all the other hashicorp products

throwaway20371 5y ago

The ironic thing is that Docker became successful only because of the tremendous hype they generated about themselves, and then failed to see the K8s hype train barreling towards them. Swarm is still what 99% of companies need when they adopt Kubernetes, and is probably 1/10th to 1/20th the cost. Somebody needs to re-invent swarm as an independent project so that we can stop investing in K8s. The cost of K8s isn't even in the complexity, it's in all the millions of side-effects of the complexity, like security, fragmentation, and specialization.

cpuguy83 5y ago
Swarm is (and always has been) an independent project: github.com/docker/swarmkit

waspight 5y ago

What is so hard about k8s? I think for example aws with fargate and all the networking there is far more complicated… i have never used swarm though.

angelzen 5y ago
k8s is just a bunch of processes working together. The downside is that if something goes wrong, it is unclear which process is faulty and how to fix it. A misconfigured DNS may take down the whole cluster, and the symptoms are that every process fails because the network is out. Difficult to backtrack to the fault source, there are a lot of configs pertaining to the network that could be faulty. On the plus side, if someone else configured the cluster correctly for you (e.g. a Cloud service like GKE), then it's a breeze to use.
geocar 5y ago
If you're talking about a hosted k8s like EKS or a toy/single-node k8s, in 2021, then nothing: k8s is much better. But if you're on-prem, and have a tonne of metal and just arrived in 2014 via a time machine, swarm was so much simpler and simpler that if you were a sysadmin who already had their own scripts -- their own jenkins-powered IC and git-hooks that built and deployed -- whatever it is you were building -- then swarm looked like a nice gradual extension of that, and k8s looked more like starting over and admitting defeat.
- waspight 5y ago
  I see your point there. I only work on the pod side, not touching any metal.
  - runlevel1 5y ago
    There's still plenty of room for improvement, but managing a K8s cluster on-prem is miles better than it was in 2014.
- Spivak 5y ago
  I mean k8s is basically an infra rewrite in any shop that was/is currently VM based. "Hey you know that fundamental unit of isolation you've built all your tooling around. Throw it all away." Don't get me wrong, I would absolutely go with k8s on any greenfield project but there's a huge huge opportunity for someone to take everyone's VM orchestration tools and quietly and semi-transparently add k8s support for gradual migrations.

rualca 5y ago

> I guess the problem was they didn't realize why Kubernetes needs to be that complicated. I guess this strongly depends on one's needs. I mean, if all you want to do is glue together a couple of nodes and have a few stacks running on them, do we really need all the bells and whistles? I mean, with Docker Swarm anyone can get ingress right from the start on any bare ones cloud provider. Deploy your Docker Swarm cluster, slap a service like Traefik, and you're done. It's a 10min job. What else does Kubernetes brings to the table?

sgk284 5y ago

K8S is great if you have Google problems, but most people don't and I think much of hype around it is much like the hype that existed a decade ago around "web scale" and "big data" and "NoSQL for everything". Docker Swarm is (with some use-case specific exceptions) more than sufficient for any shop running a few hundred nodes with a few thousand containers – and it is two orders of magnitude less complex than k8s. You can read Docker Swarm's docs end-to-end in < 30 minutes, and have a cluster running in roughly the amount of time it takes to install Docker on three boxes. Docker's failing was a failure to market Docker Swarm. Most people don't even know it exists, yet every single machine with Docker has it already! The last time I checked, they only had a single dev that was part-time allocated to supporting it. So it's not so much that Docker failed to jump on the k8s bandwagon, so much as they built a wonderful thing that they never marketed and semi-abandoned, to focus on Docker Hub and other core Docker things they thought they'd be able to more directly monetize. Fortunately, Docker Swarm remains a wonderful piece of software, even without particularly active investment, because it is a fairly complete offering for most use cases. And it seems to be growing in adoption.

bovermyer 5y ago
I was dismayed when Docker effectively (and quietly) declared that they were no longer supporting Docker Swarm. Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes. I just can't choose it over K8s, though, knowing that it's unsupported.
- sgk284 5y ago
  I've hemmed and hawed over this as well, but I believe that sometimes software can be complete and stable and not need constant iteration. As long as they add security patches and don't remove it from Docker, I'll still default to Swarm. My fallback, if Swarm is ever removed, is Hashicorp's Nomad. I've only used it in a hobby capacity, but is (at face value, at least) another wonderful piece of software. I've used k8s a bunch in production (including right now at my current employer) and just can't recommend it – it's a technical distraction for most teams.

technicolorwhat 5y ago

I think docker itself had plenty of time to innovate. They could have solved the problem long before mainstream got a hold of kubernetes. They just didn't solve the actual problems users where having at that time, which was container orchestration across multiple servers. They went very very deep with a lot of drivers and API changes and what not. If I am not mistaken docker-compose wasn't even a part of docker first before it got bundled, someone else was solving docker's same-server orchestration problems. Kubernetes is for a lot of organisations pretty difficult and way too much for a start. We used rancher for solving our problems which did a fine job, rancher got a worse when they made the move to Kubernetes. In the beginning they (docker) also removed a lot of options like cpu throttling and configuration and such that LXC had from the start. They also had their own version (a shim) of pid 1, some point at the time. And other things that made it a little painful to properly containerise. I was often very frustrated by docker and fell back on lxc. There was also something with the management of docker images like pruning and cleaning and other of much needed functionality that just didn't get included. Also something with the container registries which I a this point can't remember (maybe deleting images or authentication out of the box or something that made it hard to host yourself). Anyway I think it's failed because it failed to listen to its users and act upon them. They really had a lot of chances I think. I think they just made some wrong business decisions. I always felt they had a strong technical CTO that really was deep on the product but not on the whole pipeline from dev->x->prod workflows.

V99 5y ago
> We used rancher for solving our problems which did a fine job, rancher got a worse when they made the move to Kubernetes. (Early Rancher employee) We liked our Cattle orchestration and ease of use of 1.x as much as the next person. Hell, I still like a lot of it better. But just as this article talks about with Swarm, embracing K8s was absolutely the right move. We were the smallest of at least 4 major choices. Picking the right horse early enough and making K8s easier to use led us to a successful exit and continued relevance (now in SUSE) instead of a slow painful spiral to irrelevance like Swarm, Mesos, and others and eventual fire-sale.
- 5y ago

erulabs 5y ago

It's an incredible litmus test for developer v sysadmin. Developers think the complexity is unnecessary and so don't want to learn k8s. For sysadmins it's mostly just nice automation around all the chores we used to do with home-brewed bash scripts.

ffdf3r3rf 5y ago
As a developer, I hate k8s, but mostly because my management does not understand difference between developers and devops. They changed our job titles to devops engineers and fired sys admins. I understand k8s is complex for people who specialize in it, try imagining as a developer, you need to setup private vlan ingress. At least, on regular Linux, I know how to setup ip tables or look up 1000s of tutorial. K8s and devops has destroyed us developers' productivity, confidence, and morale.
- sho 5y ago
  Isn't it your management who have destroyed those things?
  - ffdf3r3rf 5y ago
    I don't disagree but I don't know where they are getting their marketing material where they think developers can do devops. I suspect it might some sales tactics from these Cloud/DevOps companies.
goodpoint 5y ago
I've never met a sysadmin that would prefer it over the bash scripts.
- 0des 5y ago
  What's your sample size?
  - goodpoint 5y ago

dilyevsky 5y ago

What I haven’t seen mentioned here is at least by my recollection swarm was incredibly bug ridden especially in networking layer around ‘17/18 when k8s really started pulling away

makeitdouble 5y ago

I’ve seen a number of companies going for k8s and not other solution not because they needed the complexity, but because it was first party supported. To me what killed Swarm is Google needing to spread k8s. If you are eyeing at GCE but decide you want a bit more flexibility, going for a Google backed solution gives you k8s, and Swarm isn’t even on the choice list. Choosing a third party means you’re on your own to bootstrap/make it live, or contract a vendor that might not live longer than you. That’s fine for companies who strategically choose to invest for the long term, sadly that’s not most of the companies out there.

wpietri 5y ago

> I guess the problem was they didn't realize why Kubernetes needs to be that complicated. Does it? It seems like there are three kinds of complexity involved here: the essential complexity of the total problem set, the essential complexity of a specific company's problems, and the amount of complexity experienced by a user as they try to solve those problems. Let's stipulate that Kubernetes needs to be that complicated for Google's needs. I'm not totally sure that's true, but let's pretend for now. So we'll say you're right for the first kind. But I'm still not seeing it for the other two. For modest deployment efforts, Kubernetes is a giant pain in the ass. It's just way more complicated than needed for many non-Google use cases. I tried setting up Kubernetes for a bunch of personal stuff I run and it was egregious overkill. I persisted until my home lighting system went dark because some internal Kubernetes keys expired and it decided that just not working was a good failure mode. Last time I mentioned this on HN several people said, well duh, obviously you shouldn't run it just for yourself. And that's fine; every technology has an expected operational envelope. But there are an awful lot of projects out there that have between 0.0 and 0.5 sysadmins. For them it's too complex. Kubernetes is just too complex for many people's needs. And the other side of that is complexity of experience. Modern processors are fantastically complicated. I'm on a mailing list for people in finance who micro-optimize everything to maximize response speed. Those people know incredible amounts about exactly what different processors do. And they have to. But most working developers know about 1% of that because a lot of people work very hard to hide the complexity from them. I don't think that really happened with Kubernetes. At this point, it is what it is. It's much easier to scale a technology up than down, so I expect it'll always be like this. But For whatever comes after Kubernetes, I'd like to see much more attention paid to more common use cases.

coding123 5y ago

It's really unfortunate. I am a kubernetes user today, but I LOVED swarm. K8S' main differentiator is that specific intentional "cross-cloud" features such as ingress and volume mapping that help the implementing cloud understand the deployment topology. It's basically the new PaaS but instead of working on one cloud it can work on any. I don't know that if Swarm had those features if it would really catch on like Kubernetes did. They didn't have the same budget as Google on that side of things.

SergeAx 5y ago

I beleive that wide k8s adoption is the same dead end as JS frameworks. It is not actually needed and is too complex, so it introduced it's own class of pain. VMs were a great abstraction with very little added complexity, so was Docker. K8s doesn't abstract anything away compared with Docker swarm, just adding unnecessary complexity.