6 ms·

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kuber

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.

minieggs 8y ago
I would love to see a comparison between Docker Swarm and Kubernetes. From talking to peeps I've got that Kubernetes is better for a large number of nodes? I self host all my side projects with Docker Swarm and it's been so good I haven't needed to look into other container management solutions (but I've only got eight nodes).
- piva00 8y ago
  At the moment I'd say that Kubernetes is only worthy the effort if you have a bunch of idle capacity in your nodes and some tens of machines at least. The setup can get a bit complex quite early and won't be worth the effort to manage 8 nodes, when you begin to scale to around 20 nodes running a bunch of different workloads (batch jobs, web services, etc.), can avoid provisioning on the application side, etc., then k8s begins to shine more and pay back the investment.
  - brianwawok 8y ago
    Why should node count matter? Two clicks to get 1-1000 nodes on GKE. The work is to learn the yaml syntax and the way to deploy to GKE... but most apps need to learn something about how it will be deployed (be it how to use ansible to deploy vs how to setup on k8s vs how to use serverless). But you need to do this for 1-1000 nodes, may as well just do it once...
- raesene9 8y ago
  I'd say that if you're on swarm and it's doing what you want, you probably want to stick with it. Kubernetes is more flexible and more powerful, but it's also a lot more complex. You either go with one of the managed distro's which take on some of that complexity but also reduce your flexibility, or you manage the whole thing yourself.
- jakoblorz 8y ago
  I believe this does a good job comparing without going to deep, e.g. for people just getting started on swarm/k8 https://platform9.com/blog/kubernetes-docker-swarm-compared/ https://platform9.com/blog/kubernetes-docker-swarm-compared/ Edit: updated url to a more recent version of the article
lowbloodsugar 8y ago
Mesos is a two layer system. That is what is better about it. Unfortunately, the most common second layer for reliable service execution is Marathon, and it caused us no end of trouble.
- adamtulinius 8y ago
  Spot on. The two layer system is both a blessing and a curse. Debugging problems can get really hairy because of the two layer split, and it's just not worth it if one is running Marathon as the only framework. I came to the conclusion that Mesos' biggest strength is giving people the option to write their own framework. It's not difficult, and gives you a lot of power over execution. Marathon itself appears to be very simple, but has some really weird shortcomings. For instance, it's not possible to submit a deployment and then check whether everything went well, except if you manually compare the deployment before and after (this was still true as of 1.4.x). Furthermore, Mesosphere abandoned the Marathon GUI, and I got no clue how the work of transitioning that part into the hands of volunteers are coming along, but this was essentially what tipped the scales for us. (We're currently migrating from Mesos to K8s.)
  - chrisgaun 8y ago
    While Mesos always supported a number of container orchestration layers (e.g. Netflix's Titus), you are right in that Marathon used to be the only container orchestrator for DC/OS up until 2017. Mesosphere DC/OS has had a Kubernetes package in GA since March and we've had 6 releases since then to make sure always was newest version of Kubernetes and to add features (Strict Mode). Disclaimer: I am the PM at Mesosphere.
halbritt 8y ago
Couple things: Mesos had a more sophisticated scheduler with better isolation for disk IO. It could also run arbitrary binaries in cgroups. I.e. a JVM based application without having to run it in a docker container. Uber took advantage of this by running thousands of nodes of Cassandra in DC/OS.
pageald 8y ago
Thoughts on Nomad after admittedly minimal dabbling: I pushed for using Nomad at my job without success. Managerial perceptions of Kubernetes as "the consensus" is a self-fulfilling prophecy. Nobody wants to pick a technology with a fraction of the buy-in of Kubernetes or be forced into paying boku bucks for an enterprise contract. Hashicorp's insistence on releasing premium closed-source features doesn't work when its biggest competitor is fully free and open. I found it dead simple to get up and running with Nomad, but it is (perhaps intentionally) missing a lot of features of Kubernetes. For instance, if you want load balancing and auto-scaling, you need to rig it up yourself. In K8s, you set up a service and horizontal pod autoscaling and you're done.
- fnord123 8y ago
  >be forced into paying boku bucks Boku is Japanese for "I" (cf Watashi). Beaucoup is French for "a lot".
  - dewey 8y ago
    https://www.urbandictionary.com/define.php?term=Boo%20Koo%20Bucks https://www.urbandictionary.com/define.php?term=Boo%20Koo%20...
gnur 8y ago
Having used both DC/OS (mesos & marathon) and kubernetes in production. The one thing DC/OS had absolutely nailed was the web interface and bootstrap. The DC/OS interface is brilliant as it allows you to explore all the possibilities and actually have a overview of what is happening. Also, it was a lot easier to reason about because everything is contained in a single marathon job. No need to split everything up in deployments / services / ingresses. A single JSON file is all you need for DC/OS. Less to think about. The downside of all this is that DC/OS feels like a solution for a theoretical problem, while kubernetes is the solution to practical problems.
- amazingman 8y ago
  It seems to me that running Kubernetes as a service on DC/OS might be a decent path to take. Mesosphere seem to be pushing this pretty hard, too. What did you think of DC/OS as a base platform? From an ops perspective, I’m finding a lot to like at least conceptually. Especially the idea of managing one “kind” of cluster that then manages many additional kinds for you. But I have yet to actually use it.
  - gnur 8y ago
    It sounds great, but I've always found the DC/OS marketplace not really flexible enough. I've looked at many services, but in most cases the requirements for running a service from the marketplace were for extremely high throughput situations. I wanted to experiment with arrongoDB but it required a total of 16GB memory (IIRC). Practically, I think kubernetes with helm charts gets will get you at least 90% of what DC/OS offers. Also, running Kubernetes itself is complicated enough, running it on a different scheduler will just expose you to the pains of both schedulers.
    - chrisgaun 8y ago
      Mesos means "middle" and it was designed to handle the declarative infrastructure provisioning for different schedulers... container orchestration schedulers: Aurora (Youtube/Twitter), Titus (Netflix), Marathon and Kubernetes (DC/OS).... analytics schedulers: Spark, Tensorflow, etc. The idea was always that you should manage this infrastructure just like they are managing containers. discloser: I am the PM of Kubernetes on DC/OS.
rubenbe 8y ago
I did take a look at Nomad in an attempt convince someone who was "scared" of the complexity of Kubernetes would bring to our setup. Unfortunately nice features like "Resource quotas per node" are not available in the open source version, this made Nomad a no-go.
- hardwaresofton 8y ago
  This is pretty eye-opening... If they intended to charge for extensions with functionality like that, there's no way they can compete with Kubernetes -- it has way more development happening, and features like that are already baked in to the platform... I don't know where nomad could compete to generate income, but it definitely wasn't there...
falsedan 8y ago
I haven't used Kubernetes yet since I have a Mesos scheduler that runs heavy workloads on-demand and wants to use all unused resources for a few minutes. I can't see a way to configure a k8 service to use either zero or as many pods as it possibly could, without writing something to watch the cluster's metadata and update the service's config files
- hardwaresofton 8y ago
  Kubernetes has resource limiting built in, in the form of Resource Quotas[0] and more practically (or more straight forward to use) resources for pods[1]. The system's not perfect, so it's likely possible to overcommit machines, but there are QoS controls for critical workloads. Kubernetes is built in a way that it's easy for people to write something that watches cluster metadata and perform actions (controller pattern), as such a lot of functionality is built in gradually overtime just like that. I'm not sure when resource management first came on the scene but it's been around for a while. You can also use Kubernetes to manage resources completely unrelated to Kubernetes by bringing Custom Resource Definitions ("CRDs") into play -- that's when you create a "fake" Kubernetes resource that (ex. VirtualMachine) manages some resource that's actually on the machine. The combination of CRDs and controllers to manage them is called the "Operator pattern"[2] and it's gaining a lot of hype right now as people wrestle with it conceptually but it's been around the whole time. [0]: https://kubernetes.io/docs/concepts/policy/resource-quotas/ https://kubernetes.io/docs/concepts/policy/resource-quotas/ [1]: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ https://kubernetes.io/docs/concepts/configuration/manage-com... [2]: https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
  - falsedan 8y ago
    Yeah with bare mesos I don't have to do any of that, I just accept resource offers and release them when I'm done. The scheduler decides how much it will scale, not mesos. We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster. Naturally, we have cluster-level scaling, so we add nodes until we hit our max spend per hour, and then jobs take a bit longer to complete.
JD557 8y ago
We are currently migrating from Mesos/Marathon to Kubernetes. One of the things I miss the most is the auto-refresh in the web interface. I know that's a minor issue, but after getting used to it, it's really weird having to do manual refreshes. Also, the Kubernetes UI feels too clutered, but that's probably just because I'm not used to it
- brianwawok 8y ago
  The Kubernetes UI is deprecated anyway.
  - merb 8y ago
    I think he meant Kubernetes Dashboard.
- lima 8y ago
  Try OpenShift's UI. It has auto-refresh as well as very good UX design.
- ngehani 8y ago
  WeaveCloud makes it a whole lot simpler to deploy, monitor, and manager your K8s cluster. http://cloud.weave.works http://cloud.weave.works. Give it a shot. Easy to connect to your K8s cluster and get going. Disclosure: I'm Director of Product for Weaveworks.
AaronM 8y ago
I think Nomad is a good solution when your unable to containerize your application, used it a bit at my last job in a prototype we were building to manage windows services, and it seemed to work pretty well
erikb 8y ago
Swarm: - works out of the box - has much less overhead in getting something started - tracks dependencies between services

minieggs 8y ago

I would love to see a comparison between Docker Swarm and Kubernetes. From talking to peeps I've got that Kubernetes is better for a large number of nodes? I self host all my side projects with Docker Swarm and it's been so good I haven't needed to look into other container management solutions (but I've only got eight nodes).

piva00 8y ago
At the moment I'd say that Kubernetes is only worthy the effort if you have a bunch of idle capacity in your nodes and some tens of machines at least. The setup can get a bit complex quite early and won't be worth the effort to manage 8 nodes, when you begin to scale to around 20 nodes running a bunch of different workloads (batch jobs, web services, etc.), can avoid provisioning on the application side, etc., then k8s begins to shine more and pay back the investment.
- brianwawok 8y ago
  Why should node count matter? Two clicks to get 1-1000 nodes on GKE. The work is to learn the yaml syntax and the way to deploy to GKE... but most apps need to learn something about how it will be deployed (be it how to use ansible to deploy vs how to setup on k8s vs how to use serverless). But you need to do this for 1-1000 nodes, may as well just do it once...
raesene9 8y ago
I'd say that if you're on swarm and it's doing what you want, you probably want to stick with it. Kubernetes is more flexible and more powerful, but it's also a lot more complex. You either go with one of the managed distro's which take on some of that complexity but also reduce your flexibility, or you manage the whole thing yourself.
jakoblorz 8y ago
I believe this does a good job comparing without going to deep, e.g. for people just getting started on swarm/k8 https://platform9.com/blog/kubernetes-docker-swarm-compared/ https://platform9.com/blog/kubernetes-docker-swarm-compared/ Edit: updated url to a more recent version of the article

lowbloodsugar 8y ago

Mesos is a two layer system. That is what is better about it. Unfortunately, the most common second layer for reliable service execution is Marathon, and it caused us no end of trouble.

adamtulinius 8y ago
Spot on. The two layer system is both a blessing and a curse. Debugging problems can get really hairy because of the two layer split, and it's just not worth it if one is running Marathon as the only framework. I came to the conclusion that Mesos' biggest strength is giving people the option to write their own framework. It's not difficult, and gives you a lot of power over execution. Marathon itself appears to be very simple, but has some really weird shortcomings. For instance, it's not possible to submit a deployment and then check whether everything went well, except if you manually compare the deployment before and after (this was still true as of 1.4.x). Furthermore, Mesosphere abandoned the Marathon GUI, and I got no clue how the work of transitioning that part into the hands of volunteers are coming along, but this was essentially what tipped the scales for us. (We're currently migrating from Mesos to K8s.)
- chrisgaun 8y ago
  While Mesos always supported a number of container orchestration layers (e.g. Netflix's Titus), you are right in that Marathon used to be the only container orchestrator for DC/OS up until 2017. Mesosphere DC/OS has had a Kubernetes package in GA since March and we've had 6 releases since then to make sure always was newest version of Kubernetes and to add features (Strict Mode). Disclaimer: I am the PM at Mesosphere.

halbritt 8y ago

Couple things: Mesos had a more sophisticated scheduler with better isolation for disk IO. It could also run arbitrary binaries in cgroups. I.e. a JVM based application without having to run it in a docker container. Uber took advantage of this by running thousands of nodes of Cassandra in DC/OS.

pageald 8y ago

Thoughts on Nomad after admittedly minimal dabbling: I pushed for using Nomad at my job without success. Managerial perceptions of Kubernetes as "the consensus" is a self-fulfilling prophecy. Nobody wants to pick a technology with a fraction of the buy-in of Kubernetes or be forced into paying boku bucks for an enterprise contract. Hashicorp's insistence on releasing premium closed-source features doesn't work when its biggest competitor is fully free and open. I found it dead simple to get up and running with Nomad, but it is (perhaps intentionally) missing a lot of features of Kubernetes. For instance, if you want load balancing and auto-scaling, you need to rig it up yourself. In K8s, you set up a service and horizontal pod autoscaling and you're done.

fnord123 8y ago
>be forced into paying boku bucks Boku is Japanese for "I" (cf Watashi). Beaucoup is French for "a lot".
- dewey 8y ago
  https://www.urbandictionary.com/define.php?term=Boo%20Koo%20Bucks https://www.urbandictionary.com/define.php?term=Boo%20Koo%20...

gnur 8y ago

Having used both DC/OS (mesos & marathon) and kubernetes in production. The one thing DC/OS had absolutely nailed was the web interface and bootstrap. The DC/OS interface is brilliant as it allows you to explore all the possibilities and actually have a overview of what is happening. Also, it was a lot easier to reason about because everything is contained in a single marathon job. No need to split everything up in deployments / services / ingresses. A single JSON file is all you need for DC/OS. Less to think about. The downside of all this is that DC/OS feels like a solution for a theoretical problem, while kubernetes is the solution to practical problems.

amazingman 8y ago
It seems to me that running Kubernetes as a service on DC/OS might be a decent path to take. Mesosphere seem to be pushing this pretty hard, too. What did you think of DC/OS as a base platform? From an ops perspective, I’m finding a lot to like at least conceptually. Especially the idea of managing one “kind” of cluster that then manages many additional kinds for you. But I have yet to actually use it.
- gnur 8y ago
  It sounds great, but I've always found the DC/OS marketplace not really flexible enough. I've looked at many services, but in most cases the requirements for running a service from the marketplace were for extremely high throughput situations. I wanted to experiment with arrongoDB but it required a total of 16GB memory (IIRC). Practically, I think kubernetes with helm charts gets will get you at least 90% of what DC/OS offers. Also, running Kubernetes itself is complicated enough, running it on a different scheduler will just expose you to the pains of both schedulers.
  - chrisgaun 8y ago
    Mesos means "middle" and it was designed to handle the declarative infrastructure provisioning for different schedulers... container orchestration schedulers: Aurora (Youtube/Twitter), Titus (Netflix), Marathon and Kubernetes (DC/OS).... analytics schedulers: Spark, Tensorflow, etc. The idea was always that you should manage this infrastructure just like they are managing containers. discloser: I am the PM of Kubernetes on DC/OS.

rubenbe 8y ago

I did take a look at Nomad in an attempt convince someone who was "scared" of the complexity of Kubernetes would bring to our setup. Unfortunately nice features like "Resource quotas per node" are not available in the open source version, this made Nomad a no-go.

hardwaresofton 8y ago
This is pretty eye-opening... If they intended to charge for extensions with functionality like that, there's no way they can compete with Kubernetes -- it has way more development happening, and features like that are already baked in to the platform... I don't know where nomad could compete to generate income, but it definitely wasn't there...

falsedan 8y ago

I haven't used Kubernetes yet since I have a Mesos scheduler that runs heavy workloads on-demand and wants to use all unused resources for a few minutes. I can't see a way to configure a k8 service to use either zero or as many pods as it possibly could, without writing something to watch the cluster's metadata and update the service's config files

hardwaresofton 8y ago
Kubernetes has resource limiting built in, in the form of Resource Quotas[0] and more practically (or more straight forward to use) resources for pods[1]. The system's not perfect, so it's likely possible to overcommit machines, but there are QoS controls for critical workloads. Kubernetes is built in a way that it's easy for people to write something that watches cluster metadata and perform actions (controller pattern), as such a lot of functionality is built in gradually overtime just like that. I'm not sure when resource management first came on the scene but it's been around for a while. You can also use Kubernetes to manage resources completely unrelated to Kubernetes by bringing Custom Resource Definitions ("CRDs") into play -- that's when you create a "fake" Kubernetes resource that (ex. VirtualMachine) manages some resource that's actually on the machine. The combination of CRDs and controllers to manage them is called the "Operator pattern"[2] and it's gaining a lot of hype right now as people wrestle with it conceptually but it's been around the whole time. [0]: https://kubernetes.io/docs/concepts/policy/resource-quotas/ https://kubernetes.io/docs/concepts/policy/resource-quotas/ [1]: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ https://kubernetes.io/docs/concepts/configuration/manage-com... [2]: https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
- falsedan 8y ago
  Yeah with bare mesos I don't have to do any of that, I just accept resource offers and release them when I'm done. The scheduler decides how much it will scale, not mesos. We like to create a new framework for each instance of the job (so we can track slow runs, trial-run new versions), so we can have anywhere between 0 and 50 similar frameworks all running and vying for 100% of the cluster. Naturally, we have cluster-level scaling, so we add nodes until we hit our max spend per hour, and then jobs take a bit longer to complete.

JD557 8y ago

We are currently migrating from Mesos/Marathon to Kubernetes. One of the things I miss the most is the auto-refresh in the web interface. I know that's a minor issue, but after getting used to it, it's really weird having to do manual refreshes. Also, the Kubernetes UI feels too clutered, but that's probably just because I'm not used to it

brianwawok 8y ago
The Kubernetes UI is deprecated anyway.
- merb 8y ago
  I think he meant Kubernetes Dashboard.
lima 8y ago
Try OpenShift's UI. It has auto-refresh as well as very good UX design.
ngehani 8y ago
WeaveCloud makes it a whole lot simpler to deploy, monitor, and manager your K8s cluster. http://cloud.weave.works http://cloud.weave.works. Give it a shot. Easy to connect to your K8s cluster and get going. Disclosure: I'm Director of Product for Weaveworks.

AaronM 8y ago

I think Nomad is a good solution when your unable to containerize your application, used it a bit at my last job in a prototype we were building to manage windows services, and it seemed to work pretty well

erikb 8y ago

Swarm: - works out of the box - has much less overhead in getting something started - tracks dependencies between services