8 ms·

Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes or coo

Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes or cooperates with Mesos? It feels like there's quite a bit of dense conceptual reading you have to plow through before you can even start to talk about what these things do.

ghshephard 11y ago
Add to that, how does this relate to Docker-Swarm & Core-OS Fleet. StackOverflow has at least one quick briefer, but wow, this field is growing quickly: http://stackoverflow.com/questions/27640633/docker-swarm-kubernetes-mesos-core-os-fleet http://stackoverflow.com/questions/27640633/docker-swarm-kub...
- waffle_ss 11y ago
  A couple things I've found when evaluating these: Docker Swarm seems to be the only one that supports one-off interactive containers that bind a TTY, like a Rails console (i.e. does the cluster support `docker run -it --rm busybox sh`). But its scheduling strategies[1] aren't as sophisticated as the others. Marathon doesn't support linked containers[2], so if you're using Mesos and need linked containers, you probably will want to run Kubernetes on it and use pods. [1]: https://github.com/docker/swarm/tree/b2182d080956040730cc76c31f322e637c85548a/scheduler/strategy https://github.com/docker/swarm/tree/b2182d080956040730cc76c... [2]: https://support.mesosphere.com/hc/en-us/articles/205006415-How-to-manage-applications-with-multiple-components-processes-on-Marathon- https://support.mesosphere.com/hc/en-us/articles/205006415-H...
  - throwaway1979 11y ago
    I've been playing with Swarm and like what I've seen so far. WRT Kubernetes, on a platform other than GCE, is Flannel required for networking?
    - amouat 11y ago
      I think you just need to provide a flat networking infrastructure where all nodes get their own IP and can reach each other. So you can swap flannel for project calico or your own setup if you like.
      - eepyaich 11y ago
        Yep - see http://www.projectcalico.org/calico-networking-for-kubernetes/ http://www.projectcalico.org/calico-networking-for-kubernete.... [Disclosure - I'm part of the Project Calico team.]
      - dward 11y ago
        Weave and Openvswitch are two other options. Many IaaS (gcp, aws) provide the required nobs and dials to configure this "natively" using their API so no extra SDN required if you are already running on a compatible cloud provider.
      - throwaway1979 11y ago
        I thought each host needs an entire /24 subnet, and the pods that run on the host get individual IP addresses. Am I mistaken?
        TheIronYuppie 11y ago
        No, there's a private network that's shared between all the containers running on the pods, but as long as the nodes can see each other, you're good. Full disclosure: I work at Google on Kubernetes
    - TheIronYuppie 11y ago
      Networking on AWS works fine with no additional networking as well. Overlay networking is not required if you're running within a bunch of nodes that can see each other. Only if you get more complex will you require something, and there are quite a few solutions (Flannel, Weave, Calico, etc) Full disclosure: I work at Google on Kubernetes
  - deleted 11y ago
    [deleted]
  - TheIronYuppie 11y ago
    Kubernetes does this as well: cluster/kubectl.sh exec pod_name -ti bash Full disclosure: I work at Google on Kubernetes
    - waffle_ss 11y ago
      Awesome, good to know. Based on this issue[1] I didn't think it did. [1]: https://github.com/GoogleCloudPlatform/kubernetes/issues/1521 https://github.com/GoogleCloudPlatform/kubernetes/issues/152...
WestCoastJustin 11y ago
At their most basic level, Kubernetes [1] and Mesos [2] both use a client/server type architecture, where you install a client on many compute nodes, then the server part farms out jobs (in the form of containers) to these client compute nodes. Mesos does not do scheduling, and is pretty low-level in terms of interfaces, so you would typically run some type of software which talks to it via API, like Marathon [3], Aurora [4], or Hadoop [5]. Then Marathon/Aurora/Hadoop/etc tells Mesos to farm out compute jobs to these end client nodes (aka schedules). Complexity can quickly go up depending on your hosting environment, scale, and HA requirements. There is actually a really good high-level overview diagram of what a mesos/hadoop/zookeeper setup looks like here [6]. The stack looks something like this for Kubernetes/Mesos (top down): - Kubernetes and Mesos (client/server packages depending on node type) - Docker (container engine) - OS (Ubuntu/RHEL/etc) What are some use-cases? - you have more work than can fit into one server - need to distribute load across N+ nodes - Google heavily uses containers (not k8s, but that inspired these patterns) - Gmail/Search/etc all run in containers [7] - Apple, Twitter, and Airbnb are running Mesos today [8, 9] There are a bunch of revolving services, like: - distributed key/values stores (etcd/zookeeper) - load balancers - image registries - user interfaces - cli tools - logging/monitoring/alerting - etc But, to answer your question, the main difference between Kubernetes and Mesos, is that Kubernetes offers an opinionated workflow, built-in scheduler, and patterns for how containers are deployed into this cluster of compute notes. The pattern is baked in from the start via Pods, Labels, Services, and Replication Controllers. It also helps to know that Kubernetes comes from Google, where they have been running containers in-house, so much of this workflow (pods, services, replication controllers, etc), comes from their internal use-cases. That's the 10,000 foot view. [1] https://github.com/GoogleCloudPlatform/kubernetes https://github.com/GoogleCloudPlatform/kubernetes [2] http://mesos.apache.org/ http://mesos.apache.org/ [3] https://mesosphere.github.io/marathon/ https://mesosphere.github.io/marathon/ [4] http://aurora.apache.org/ http://aurora.apache.org/ [5] https://hadoop.apache.org/ https://hadoop.apache.org/ [6] http://mesos.apache.org/assets/img/documentation/architecture3.jpg http://mesos.apache.org/assets/img/documentation/architectur... [7] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [8] http://www.infoq.com/news/2015/05/mesos-powers-apple-siri http://www.infoq.com/news/2015/05/mesos-powers-apple-siri [9] https://www.youtube.com/watch?v=E4lxX6epM_U https://www.youtube.com/watch?v=E4lxX6epM_U
mml 11y ago
Kubernetes can run on Mesos as a Mesos service. Thereby giving you a more opinionated layout for your compute cluster, while running other peer Mesos services, such as Hadoop, Chronos And Marathon. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
- ninkendo 11y ago
  I'm afraid answers like this actually make the confusion problem worse (nothing against your comment, just an observation in general.) If you're confused about the differences between similar-sounding products X and Y, the fact that "X runs on Y" or "Y supports X" has never made the situation any better, it only makes the line between X and Y even more blurred. I think this is especially true of Mesos, because people have a tendency to attribute qualities to Mesos that are actually qualities of a particular Mesos framework like Marathon or Aurora. As it is, Mesos is more of an SDK than anything, giving you the tools to write an orchestration system. It comes with built-in ways to communicate with nodes over a message bus, the ability to look at your nodes as resources, etc... but all of the scheduling logic is up to the frameworks themselves. I think Mesos has a perception problem because of this. They want to build up hype about what mesos is and can do, so they claim things like Mesos being able to schedule docker images and keep them running, etc... but that's really the job of something like Marathon that runs as a Mesos framework. But if they didn't claim such things, Mesos wouldn't seem very compelling. To me, the biggest benefit of Mesos is what the gain would be if every datacenter scheduler was a Mesos framework (yarn/spark/kubernetes/marathon/fleet/swarm/aurora, etc), and Mesos was only used to maintain multitenancy on the same hardware. That's where the real advantages come from... if you want to try Kubernetes you shouldn't have to dedicate hardware to it, you should just install it on your existing mesos cluster that is already running the rest of your stuff. In this respect, mesos is only useful insofar as all the big cluster managers use it as their underlying substrate.
  - tlrobinson 11y ago
    "X runs on Y" at least implies Y is "lower level" than X. As I understand it, Mesos is analogous to an operating system kernel for your cluster, while Kubernetes is a CaaS (containers-as-a-service) layer on top.
MehdiEG 11y ago
To add to discussion, I've found this interview with Malte Schwarzkopf to be quite enlighning to understand how Omega, Kubernetes and Mesos fit together: https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distributed-systems-orchestration-in-the-modern-data-center/ https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distr...
bkeroack 11y ago
To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application. - Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running. - Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on. - By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes. People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations. You can now cut me a check for 50% of the consulting revenue you get from this information. :) 1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.
- davidooo 11y ago
  The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.
  - bkeroack 11y ago
    Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
    - 11y ago

ghshephard 11y ago

Add to that, how does this relate to Docker-Swarm & Core-OS Fleet. StackOverflow has at least one quick briefer, but wow, this field is growing quickly: http://stackoverflow.com/questions/27640633/docker-swarm-kubernetes-mesos-core-os-fleet http://stackoverflow.com/questions/27640633/docker-swarm-kub...

waffle_ss 11y ago
A couple things I've found when evaluating these: Docker Swarm seems to be the only one that supports one-off interactive containers that bind a TTY, like a Rails console (i.e. does the cluster support `docker run -it --rm busybox sh`). But its scheduling strategies[1] aren't as sophisticated as the others. Marathon doesn't support linked containers[2], so if you're using Mesos and need linked containers, you probably will want to run Kubernetes on it and use pods. [1]: https://github.com/docker/swarm/tree/b2182d080956040730cc76c31f322e637c85548a/scheduler/strategy https://github.com/docker/swarm/tree/b2182d080956040730cc76c... [2]: https://support.mesosphere.com/hc/en-us/articles/205006415-How-to-manage-applications-with-multiple-components-processes-on-Marathon- https://support.mesosphere.com/hc/en-us/articles/205006415-H...
- throwaway1979 11y ago
  I've been playing with Swarm and like what I've seen so far. WRT Kubernetes, on a platform other than GCE, is Flannel required for networking?
  - amouat 11y ago
    I think you just need to provide a flat networking infrastructure where all nodes get their own IP and can reach each other. So you can swap flannel for project calico or your own setup if you like.
    - eepyaich 11y ago
      Yep - see http://www.projectcalico.org/calico-networking-for-kubernetes/ http://www.projectcalico.org/calico-networking-for-kubernete.... [Disclosure - I'm part of the Project Calico team.]
    - dward 11y ago
      Weave and Openvswitch are two other options. Many IaaS (gcp, aws) provide the required nobs and dials to configure this "natively" using their API so no extra SDN required if you are already running on a compatible cloud provider.
    - throwaway1979 11y ago
      I thought each host needs an entire /24 subnet, and the pods that run on the host get individual IP addresses. Am I mistaken?
      - TheIronYuppie 11y ago
        No, there's a private network that's shared between all the containers running on the pods, but as long as the nodes can see each other, you're good. Full disclosure: I work at Google on Kubernetes
  - TheIronYuppie 11y ago
    Networking on AWS works fine with no additional networking as well. Overlay networking is not required if you're running within a bunch of nodes that can see each other. Only if you get more complex will you require something, and there are quite a few solutions (Flannel, Weave, Calico, etc) Full disclosure: I work at Google on Kubernetes
- deleted 11y ago
  [deleted]
- TheIronYuppie 11y ago
  Kubernetes does this as well: cluster/kubectl.sh exec pod_name -ti bash Full disclosure: I work at Google on Kubernetes
  - waffle_ss 11y ago
    Awesome, good to know. Based on this issue[1] I didn't think it did. [1]: https://github.com/GoogleCloudPlatform/kubernetes/issues/1521 https://github.com/GoogleCloudPlatform/kubernetes/issues/152...

WestCoastJustin 11y ago

At their most basic level, Kubernetes [1] and Mesos [2] both use a client/server type architecture, where you install a client on many compute nodes, then the server part farms out jobs (in the form of containers) to these client compute nodes. Mesos does not do scheduling, and is pretty low-level in terms of interfaces, so you would typically run some type of software which talks to it via API, like Marathon [3], Aurora [4], or Hadoop [5]. Then Marathon/Aurora/Hadoop/etc tells Mesos to farm out compute jobs to these end client nodes (aka schedules). Complexity can quickly go up depending on your hosting environment, scale, and HA requirements. There is actually a really good high-level overview diagram of what a mesos/hadoop/zookeeper setup looks like here [6]. The stack looks something like this for Kubernetes/Mesos (top down): - Kubernetes and Mesos (client/server packages depending on node type) - Docker (container engine) - OS (Ubuntu/RHEL/etc) What are some use-cases? - you have more work than can fit into one server - need to distribute load across N+ nodes - Google heavily uses containers (not k8s, but that inspired these patterns) - Gmail/Search/etc all run in containers [7] - Apple, Twitter, and Airbnb are running Mesos today [8, 9] There are a bunch of revolving services, like: - distributed key/values stores (etcd/zookeeper) - load balancers - image registries - user interfaces - cli tools - logging/monitoring/alerting - etc But, to answer your question, the main difference between Kubernetes and Mesos, is that Kubernetes offers an opinionated workflow, built-in scheduler, and patterns for how containers are deployed into this cluster of compute notes. The pattern is baked in from the start via Pods, Labels, Services, and Replication Controllers. It also helps to know that Kubernetes comes from Google, where they have been running containers in-house, so much of this workflow (pods, services, replication controllers, etc), comes from their internal use-cases. That's the 10,000 foot view. [1] https://github.com/GoogleCloudPlatform/kubernetes https://github.com/GoogleCloudPlatform/kubernetes [2] http://mesos.apache.org/ http://mesos.apache.org/ [3] https://mesosphere.github.io/marathon/ https://mesosphere.github.io/marathon/ [4] http://aurora.apache.org/ http://aurora.apache.org/ [5] https://hadoop.apache.org/ https://hadoop.apache.org/ [6] http://mesos.apache.org/assets/img/documentation/architecture3.jpg http://mesos.apache.org/assets/img/documentation/architectur... [7] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [8] http://www.infoq.com/news/2015/05/mesos-powers-apple-siri http://www.infoq.com/news/2015/05/mesos-powers-apple-siri [9] https://www.youtube.com/watch?v=E4lxX6epM_U https://www.youtube.com/watch?v=E4lxX6epM_U

mml 11y ago

Kubernetes can run on Mesos as a Mesos service. Thereby giving you a more opinionated layout for your compute cluster, while running other peer Mesos services, such as Hadoop, Chronos And Marathon. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

ninkendo 11y ago
I'm afraid answers like this actually make the confusion problem worse (nothing against your comment, just an observation in general.) If you're confused about the differences between similar-sounding products X and Y, the fact that "X runs on Y" or "Y supports X" has never made the situation any better, it only makes the line between X and Y even more blurred. I think this is especially true of Mesos, because people have a tendency to attribute qualities to Mesos that are actually qualities of a particular Mesos framework like Marathon or Aurora. As it is, Mesos is more of an SDK than anything, giving you the tools to write an orchestration system. It comes with built-in ways to communicate with nodes over a message bus, the ability to look at your nodes as resources, etc... but all of the scheduling logic is up to the frameworks themselves. I think Mesos has a perception problem because of this. They want to build up hype about what mesos is and can do, so they claim things like Mesos being able to schedule docker images and keep them running, etc... but that's really the job of something like Marathon that runs as a Mesos framework. But if they didn't claim such things, Mesos wouldn't seem very compelling. To me, the biggest benefit of Mesos is what the gain would be if every datacenter scheduler was a Mesos framework (yarn/spark/kubernetes/marathon/fleet/swarm/aurora, etc), and Mesos was only used to maintain multitenancy on the same hardware. That's where the real advantages come from... if you want to try Kubernetes you shouldn't have to dedicate hardware to it, you should just install it on your existing mesos cluster that is already running the rest of your stuff. In this respect, mesos is only useful insofar as all the big cluster managers use it as their underlying substrate.
- tlrobinson 11y ago
  "X runs on Y" at least implies Y is "lower level" than X. As I understand it, Mesos is analogous to an operating system kernel for your cluster, while Kubernetes is a CaaS (containers-as-a-service) layer on top.

MehdiEG 11y ago

To add to discussion, I've found this interview with Malte Schwarzkopf to be quite enlighning to understand how Omega, Kubernetes and Mesos fit together: https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distributed-systems-orchestration-in-the-modern-data-center/ https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distr...

bkeroack 11y ago

To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application. - Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running. - Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on. - By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes. People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations. You can now cut me a check for 50% of the consulting revenue you get from this information. :) 1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.

davidooo 11y ago
The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.
- bkeroack 11y ago
  Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
  - 11y ago