4 ms·

>Kubernetes allows us to run workloads (Flink, Cassandra, Spark, and Kafka, among others) that were once difficult to manage under Mesos (due to local state req

>Kubernetes allows us to run workloads (Flink, Cassandra, Spark, and Kafka, among others) that were once difficult to manage under Mesos (due to local state requirements). Just wanted to clarify that running Cassandra & Kafka on Mesos is much easier now with the DC/OS Commons SDK.[1] Spark has always been supported on Mesos.[2] [1]https://github.com/mesosphere/dcos-commons https://github.com/mesosphere/dcos-commons Cassandra:https://docs.d2iq.com/mesosphere/dcos/services/cassandra/2.7.0-3.11.4/ https://docs.d2iq.com/mesosphere/dcos/services/cassandra/2.7... Kafka:https://docs.d2iq.com/mesosphere/dcos/services/kafka/2.8.0-2.3.0/ https://docs.d2iq.com/mesosphere/dcos/services/kafka/2.8.0-2... [2]https://docs.d2iq.com/mesosphere/dcos/services/spark/2.9.0-2.4.3/ https://docs.d2iq.com/mesosphere/dcos/services/spark/2.9.0-2...

elvinyung 7y ago
Yes -- in fact, the original Spark implementation ran on Mesos only, as per the original RDD paper by Zaharia et al.: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf https://www.usenix.org/system/files/conference/nsdi12/nsdi12...
ssk2 7y ago
Small point of clarification: the DC/OS SDK works well for these applications when running on DC/OS. It did not work well when run on open source Mesos, which is what Yelp uses.
- perfect_kiss 7y ago
  DC/OS is also open source, Apache 2.0, same as Mesos. In fact DC/OS includes open source Mesos with no modifications, what they add is packaged bunch of different API providers running as Mesos apps: Marathon, DNS, etc., and also a somewhat dumb installer. Auto-scaling was long possible for Marathon/Mesos, and is described at DC/OS website, though does not involve any of DC/OS additionals: https://docs.d2iq.com/mesosphere/dcos/2.0/tutorials/autoscaling/ https://docs.d2iq.com/mesosphere/dcos/2.0/tutorials/autoscal... . In my org, we run open source Marathon/Mesos deployments for years without any DC/OS additionals, and also a slightly modified versions of tooling below: https://github.com/mesosphere/marathon-autoscale https://github.com/mesosphere/marathon-autoscale https://github.com/mesosphere/marathon-lb-autoscale https://github.com/mesosphere/marathon-lb-autoscale There are also available several open-source auto-scalers for Marathon/Mesos apart from these two. This new one from Yelp seems to be promising, since it's battle tested within organization which has many kinds of different workloads, so I would give it a try for some new cluster.

elvinyung 7y ago

Yes -- in fact, the original Spark implementation ran on Mesos only, as per the original RDD paper by Zaharia et al.: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf https://www.usenix.org/system/files/conference/nsdi12/nsdi12...

ssk2 7y ago

Small point of clarification: the DC/OS SDK works well for these applications when running on DC/OS. It did not work well when run on open source Mesos, which is what Yelp uses.

perfect_kiss 7y ago
DC/OS is also open source, Apache 2.0, same as Mesos. In fact DC/OS includes open source Mesos with no modifications, what they add is packaged bunch of different API providers running as Mesos apps: Marathon, DNS, etc., and also a somewhat dumb installer. Auto-scaling was long possible for Marathon/Mesos, and is described at DC/OS website, though does not involve any of DC/OS additionals: https://docs.d2iq.com/mesosphere/dcos/2.0/tutorials/autoscaling/ https://docs.d2iq.com/mesosphere/dcos/2.0/tutorials/autoscal... . In my org, we run open source Marathon/Mesos deployments for years without any DC/OS additionals, and also a slightly modified versions of tooling below: https://github.com/mesosphere/marathon-autoscale https://github.com/mesosphere/marathon-autoscale https://github.com/mesosphere/marathon-lb-autoscale https://github.com/mesosphere/marathon-lb-autoscale There are also available several open-source auto-scalers for Marathon/Mesos apart from these two. This new one from Yelp seems to be promising, since it's battle tested within organization which has many kinds of different workloads, so I would give it a try for some new cluster.