3 ms·

To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not po

To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application. - Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running. - Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on. - By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes. People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations. You can now cut me a check for 50% of the consulting revenue you get from this information. :) 1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.

davidooo 11y ago
The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.
- bkeroack 11y ago
  Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
  - davidooo 11y ago
    Currently resource requirements are specified only on containers, not Pods. The requirements for the Pod are computed by adding up the requirements of the containers within the Pod. To be more concrete: Within the PodSpec type that you linked to, there is a field of type []Container. Within the Container type there is a field called Resources which is of type ResourceRequirements. ResourceRequirements lets you specify resource requirements of the container. The resource requirements of the Pod are computed by adding up the resource requirements of the containers that run within the Pod. In addition to resource-based scheduling, we also support "label selectors" which allows you to label nodes with key/value pairs and then say that a Pod should only run on nodes with particular labels. That's specified in the NodeSelector field of the PodSpec (which you linked to).
    - bkeroack 11y ago
      Fair enough! :) I overlooked that aspect of the Container type, obviously.
  - deleted 11y ago
    [deleted]
  - bcbroussard 11y ago
    Here's the doc for resource constraints, called "limits" in Kubernetes. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/limitrange/README.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... The Kubernetes scheduler looks at "fit" and "resource availability" to determine the node a pod will run on. Nodes can also have labels like "high-mem" or "ssd", so you can request a particular type of server (via the the nodeSelector field). More details are in the link above.
    - davidooo 11y ago
      The page you linked to describes a slightly different feature, namely the ability to restrict and override the resource requirements of Pods at the time they are submitted to the system. So it's part of the admission control system, not part of the scheduling. The documentation on resource-based scheduling is at https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/compute-resources.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
merb 11y ago
What you forgot is that it's not only easy to get started, you could also start REALLY REALLY small which doesn't work with ALL the other solutions. Kubernetes Master could run on a node with 512mb memory and it still works great. Your cluster doesn't die if the master dies it won't reschedule etc but thats ok mostly. the only thing what you need is enough etcd nodes to keep it up and running.
SEJeff 11y ago
Mesosphere is a company, not a piece of software. I think you're referring to Mesosphere's dcos[1]. DCOS is some really nice packaging for marathon, chronos, etc, with a nice cli tool for downloading and installing new frameworks onto mesos. Personally, I find using Aurora a lot nicer. That being said, I do think the kubernetes-mesos gives you far and above the best of both worlds. You get the developer story of kubernetes, with the ops story of mesos. From an ops perspective, k8s is a bit clunky. I was actually shocked when I found out after bringing a new kubelet (worker node) online, you have to also update a service on the master. This was in a power training class on Kubernetes at this year's Redhat Summit in Boston. Really underwhelmed with the complexity of k8s compared to mesos, but they aren't an apples to apples comparison. [1] https://mesosphere.com/product/ https://mesosphere.com/product/
- bkeroack 11y ago
  > I was actually shocked when I found out after bringing a new kubelet (worker node) online, you have to also update a service on the master On AWS that has not been my experience. Nodes can be brought up dynamically and they register themselves with the master.
  - deleted 11y ago
    [deleted]
  - SEJeff 11y ago
    Can't speak for the cloud. When running on bare metal, in high availability mode, you need to edit the controller manager config, /etc/kubernetes/controller-manager, and update a line called KUBELET_ADDRESSES. If you don't, they won't be in the output of: kubectl get minions Or usable at all. Really frustrating to me.

davidooo 11y ago

The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.

bkeroack 11y ago
Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
- davidooo 11y ago
  Currently resource requirements are specified only on containers, not Pods. The requirements for the Pod are computed by adding up the requirements of the containers within the Pod. To be more concrete: Within the PodSpec type that you linked to, there is a field of type []Container. Within the Container type there is a field called Resources which is of type ResourceRequirements. ResourceRequirements lets you specify resource requirements of the container. The resource requirements of the Pod are computed by adding up the resource requirements of the containers that run within the Pod. In addition to resource-based scheduling, we also support "label selectors" which allows you to label nodes with key/value pairs and then say that a Pod should only run on nodes with particular labels. That's specified in the NodeSelector field of the PodSpec (which you linked to).
  - bkeroack 11y ago
    Fair enough! :) I overlooked that aspect of the Container type, obviously.
- deleted 11y ago
  [deleted]
- bcbroussard 11y ago
  Here's the doc for resource constraints, called "limits" in Kubernetes. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/limitrange/README.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... The Kubernetes scheduler looks at "fit" and "resource availability" to determine the node a pod will run on. Nodes can also have labels like "high-mem" or "ssd", so you can request a particular type of server (via the the nodeSelector field). More details are in the link above.
  - davidooo 11y ago
    The page you linked to describes a slightly different feature, namely the ability to restrict and override the resource requirements of Pods at the time they are submitted to the system. So it's part of the admission control system, not part of the scheduling. The documentation on resource-based scheduling is at https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/compute-resources.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

merb 11y ago

What you forgot is that it's not only easy to get started, you could also start REALLY REALLY small which doesn't work with ALL the other solutions. Kubernetes Master could run on a node with 512mb memory and it still works great. Your cluster doesn't die if the master dies it won't reschedule etc but thats ok mostly. the only thing what you need is enough etcd nodes to keep it up and running.

SEJeff 11y ago

Mesosphere is a company, not a piece of software. I think you're referring to Mesosphere's dcos[1]. DCOS is some really nice packaging for marathon, chronos, etc, with a nice cli tool for downloading and installing new frameworks onto mesos. Personally, I find using Aurora a lot nicer. That being said, I do think the kubernetes-mesos gives you far and above the best of both worlds. You get the developer story of kubernetes, with the ops story of mesos. From an ops perspective, k8s is a bit clunky. I was actually shocked when I found out after bringing a new kubelet (worker node) online, you have to also update a service on the master. This was in a power training class on Kubernetes at this year's Redhat Summit in Boston. Really underwhelmed with the complexity of k8s compared to mesos, but they aren't an apples to apples comparison. [1] https://mesosphere.com/product/ https://mesosphere.com/product/

bkeroack 11y ago
> I was actually shocked when I found out after bringing a new kubelet (worker node) online, you have to also update a service on the master On AWS that has not been my experience. Nodes can be brought up dynamically and they register themselves with the master.
- deleted 11y ago
  [deleted]
- SEJeff 11y ago
  Can't speak for the cloud. When running on bare metal, in high availability mode, you need to edit the controller manager config, /etc/kubernetes/controller-manager, and update a line called KUBELET_ADDRESSES. If you don't, they won't be in the output of: kubectl get minions Or usable at all. Really frustrating to me.