4 ms·

I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each ope

I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go. Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscaling that is annoying to calculate in Mesos. (The Mesos master has no idea how many pods each framework _wants_ to launch, so we ended up writing code that knows how to deduce the resource demand from each framework's API or UI.) On the other side of the same coin, the framework in Mesos has no idea the total resources available to the cluster - it has to wait to be offered each box's resources in turn. This usually means frameworks either: a) accept the first offer which matches the basic requirements for a task, even if there are better places to run that task, or b) accept every single offer, under the assumption that they're the only framework on the cluster, and then implement their own scheduler on top of this pool of resources. (I call this the "monoframework" approach.) The former approach is how Mesos is supposed to work, but can lead to all sorts of bad outcomes, e.g. all the copies of a Marathon service running on the same box, or Spark having to use timeouts to know when it should give up waiting for offers if it doesn't get enough right away. The latter approach can lead to better placements, but defeats the purpose of the two-level scheduler, as no other frameworks can use the resources that aren't being used by the monoframework. Under Kubernetes, any operator can query the state of the cluster and make informed decisions about what to request. 1. https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...

jaaron 6y ago
My frame of reference for the argument about Mesos vs k8s is from earlier in the timeline when the k8s was up and coming and just starting to compete with Mesos. Just as in my analogy with Rails: at it's inception, it lacked a bunch of features, but it was accessible. Years later, Rails had many of those features. k8s is the same. Operators didn't come around until late 2016 by CoreOS and even then they weren't widely adopted. It wasn't until after RedHat bought CoreOS and pushed operators as a pattern into k8s that the pattern took off. As it is, the k8s version of the operator framework only went 1.0 last year. Finally, I should have pointed it out in my original comment, the Mesos vs k8s comparison isn't perfect apples and apples. Mesos is just a component in a stack, whereas k8s is effectively a full stack. Again, Rails was pitched as an opinionated, batteries-included framework compared to many of the more focused frameworks that, for their niche, may have been better, but the convenience of having everything in one packages won out.

jaaron 6y ago

My frame of reference for the argument about Mesos vs k8s is from earlier in the timeline when the k8s was up and coming and just starting to compete with Mesos. Just as in my analogy with Rails: at it's inception, it lacked a bunch of features, but it was accessible. Years later, Rails had many of those features. k8s is the same. Operators didn't come around until late 2016 by CoreOS and even then they weren't widely adopted. It wasn't until after RedHat bought CoreOS and pushed operators as a pattern into k8s that the pattern took off. As it is, the k8s version of the operator framework only went 1.0 last year. Finally, I should have pointed it out in my original comment, the Mesos vs k8s comparison isn't perfect apples and apples. Mesos is just a component in a stack, whereas k8s is effectively a full stack. Again, Rails was pitched as an opinionated, batteries-included framework compared to many of the more focused frameworks that, for their niche, may have been better, but the convenience of having everything in one packages won out.