5 ms·

I recently evaluated all 3 solutions. Here is how I see it after testing the waters: - want something simple that works today? Docker Swarm - want something a

I recently evaluated all 3 solutions. Here is how I see it after testing the waters: - want something simple that works today? Docker Swarm - want something amazingly flexible? Kubernetes - already use Mesos or DC/OS? Marathon/Mesos This article from Mesosphere is interesting and gives a good overview, but it downplays advantages of Swarm and Kubernetes and clearly highlights Mesos: Docker has 4 bullet points, Kubernetes has 3 and Mesos has 5. In addition, Marathon is meant to work on Mesosphere DC/OS and not on other operating systems. For instance, the "Virtual IP" [1] feature only works on DC/OS from what I can tell. This confused me because this feature appears on the UI even if you run Ubuntu, it just doesn't work in that setup. Docker Swarm seems to have a very comparable feature set, but doesn't care what OS you run. There is an Mesos plugin for Kubernetes [2] but it looks unmaintained, so this sentence seems a bit off: "Mesos could even run Kubernetes or other container orchestrators, though a public integration is not yet available." This also goes to show how important community support and commercial backing is and Kubernetes is the clear winner here. At the end, you can read this: "want all of this portable across cloud providers and/or datacenters, then Mesos (or our own Mesos distribution, Mesosphere DC/OS)". With Docker Swarm, you can use multiple cloud providers. Docker Swarm does not care about what servers you use. It encrypts traffic between nodes and needs Docker. That's it. It is cloud agnostic. I'm not sure what the story is with Kubernetes here. Kubernetes is a huge beast: very flexible and powerful, but a high upfront setup cost when you want to manage everything yourself. [1] https://dcos.io/docs/1.9/networking/load-balancing-vips/ https://dcos.io/docs/1.9/networking/load-balancing-vips/ [2] https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos

davidopp_ 9y ago
Kubernetes runs on GCP, AWS, and Azure, and in various on-prem configurations (including on OpenStack). Disclosure: I work on Kubernetes at Google.
- conradk 9y ago
  Hello David, thanks for your input here! I was wondering specifically about a hybrid cloud / multi-az setup. Does Kubernetes support that?
  - sisk 9y ago
    I'm not David but the answer is yes. After spinning up multiple discrete clusters, you can orchestrate across them by deploying the federation control plane. https://kubernetes.io/docs/concepts/cluster-administration/federation/ https://kubernetes.io/docs/concepts/cluster-administration/f...
drdaeman 9y ago
Uh. I'm skeptic about Swarm being something that "works today". I really want to use it and spent some time experimenting, but failed to get it working. It could work for a toy webapp project, but I believe is insufficient for anything complicated. There are a lot of things that one would expect to have but that are yet unsolved. Or I'm just unaware that there is a solution (or the solution didn't fit my personal requirements). 1. Networking options are really limited. Built-in ingress loses of the originating IP address, cannot bind ports only on specific nodes (something that may have partially fixed this was implemented in 17.06 with --network=host option, but I haven't found much documentation), cannot prefer same-node containers (which is important if you have many microservices, as the latency adds up quickly) and more. For ingress, if one's lucky to have only HTTP traffic (I don't), they can use something like Traefik. But they'll need to run LBs on a manager nodes which isn't really a smart thing to do, as Swarm is said to be sensitive to overloaded managers. Or one can just go for external LBs, like CloudFlare or whatever Amazon/Google/Microsoft offers. If one needs raw TCP, UDP or other IP traffic - I think they'd better completely ignore built-in service discovery and LBs and go for something external, like etcd+haproxy/nginx, outside of the Swarm (on nodes' host OS). It has to be completely manual setup (okay, I meant Ansible/Puppet/Chef/Salt/etc). While it's possible to run LBs with Docker (non-Swarm containers, if the Swarm network is attachable), etcd is just not designed to auto-deploy on Swarm: its design explicitly disallows one to just `docker service create --name etcd my/custom/etcd && docker service scale etcd=5`, you'll need to set up every node by hand. I believe Consul is better in this regard but haven't tried it yet. 2. I haven't figured out how to have a persistent storage that follows the containers. If a node dies, Swarm would spawn new containers on other nodes, but no chance to have even a slightly dated snapshot. There was something called Flocker that looked like a solution, but it's essentially dead (despite the revival attempts). GlusterFS is an option I know about, but it's really sensitive to latency. Databases are even more tricky, unless one's bold to use something fancy like CockroachDB (I had enough subtle issues with RethinkDB to be wary about bleeding edge stuff). Maybe I'm just too stupid, but I failed to grok dynamic PostgreSQL multi-master BDR setup, so my DB is still SPOF with some WAL streaming replication manually-activated failovers. 3. Secrets look like a nice addition, but they're best avoided. They're immutable and you have to recreate the container to switch them. If you have any services that have many user-initiated long-living connections (e.g. IRC, XMPP, Websockets or media streaming), this would makes secrets basically unusable for anything that could be rotated, like TLS certificates. Unless you can drop all your users every now and then. 4. Logging was quite messy, but they've sorted it out with 17.06. (As for the K8S - it solves most of the issues, but I got my share of issues with Rancher, so I'm really wary about having any complexity in the core. There's already a beast called Linux kernel down there, and $deity have mercy on those who have to debug its oopses. If a behemoth - I mean, K8s - decides to misbehave, I expect to have a really bad time trying to keep things afloat. Even Swarm mode is fairly complex black box binary - but at least I can try debugging it.)

davidopp_ 9y ago

Kubernetes runs on GCP, AWS, and Azure, and in various on-prem configurations (including on OpenStack). Disclosure: I work on Kubernetes at Google.

conradk 9y ago
Hello David, thanks for your input here! I was wondering specifically about a hybrid cloud / multi-az setup. Does Kubernetes support that?
- sisk 9y ago
  I'm not David but the answer is yes. After spinning up multiple discrete clusters, you can orchestrate across them by deploying the federation control plane. https://kubernetes.io/docs/concepts/cluster-administration/federation/ https://kubernetes.io/docs/concepts/cluster-administration/f...

drdaeman 9y ago

Uh. I'm skeptic about Swarm being something that "works today". I really want to use it and spent some time experimenting, but failed to get it working. It could work for a toy webapp project, but I believe is insufficient for anything complicated. There are a lot of things that one would expect to have but that are yet unsolved. Or I'm just unaware that there is a solution (or the solution didn't fit my personal requirements). 1. Networking options are really limited. Built-in ingress loses of the originating IP address, cannot bind ports only on specific nodes (something that may have partially fixed this was implemented in 17.06 with --network=host option, but I haven't found much documentation), cannot prefer same-node containers (which is important if you have many microservices, as the latency adds up quickly) and more. For ingress, if one's lucky to have only HTTP traffic (I don't), they can use something like Traefik. But they'll need to run LBs on a manager nodes which isn't really a smart thing to do, as Swarm is said to be sensitive to overloaded managers. Or one can just go for external LBs, like CloudFlare or whatever Amazon/Google/Microsoft offers. If one needs raw TCP, UDP or other IP traffic - I think they'd better completely ignore built-in service discovery and LBs and go for something external, like etcd+haproxy/nginx, outside of the Swarm (on nodes' host OS). It has to be completely manual setup (okay, I meant Ansible/Puppet/Chef/Salt/etc). While it's possible to run LBs with Docker (non-Swarm containers, if the Swarm network is attachable), etcd is just not designed to auto-deploy on Swarm: its design explicitly disallows one to just `docker service create --name etcd my/custom/etcd && docker service scale etcd=5`, you'll need to set up every node by hand. I believe Consul is better in this regard but haven't tried it yet. 2. I haven't figured out how to have a persistent storage that follows the containers. If a node dies, Swarm would spawn new containers on other nodes, but no chance to have even a slightly dated snapshot. There was something called Flocker that looked like a solution, but it's essentially dead (despite the revival attempts). GlusterFS is an option I know about, but it's really sensitive to latency. Databases are even more tricky, unless one's bold to use something fancy like CockroachDB (I had enough subtle issues with RethinkDB to be wary about bleeding edge stuff). Maybe I'm just too stupid, but I failed to grok dynamic PostgreSQL multi-master BDR setup, so my DB is still SPOF with some WAL streaming replication manually-activated failovers. 3. Secrets look like a nice addition, but they're best avoided. They're immutable and you have to recreate the container to switch them. If you have any services that have many user-initiated long-living connections (e.g. IRC, XMPP, Websockets or media streaming), this would makes secrets basically unusable for anything that could be rotated, like TLS certificates. Unless you can drop all your users every now and then. 4. Logging was quite messy, but they've sorted it out with 17.06. (As for the K8S - it solves most of the issues, but I got my share of issues with Rancher, so I'm really wary about having any complexity in the core. There's already a beast called Linux kernel down there, and $deity have mercy on those who have to debug its oopses. If a behemoth - I mean, K8s - decides to misbehave, I expect to have a really bad time trying to keep things afloat. Even Swarm mode is fairly complex black box binary - but at least I can try debugging it.)