4 ms·
I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overg
by jaaron 6y ago
I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning:
k8s is the Rails of the cloud.
Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterprise web frameworks, Rails felt like a toy. It didn't have the features, robustness, safety, and scalability of "proper" frameworks.
What did Rails do? It was easy to get started and it hid a lot of the boring and painful work of web frameworks at the time.
Through the sheer force of will of a massive community, Rails grew up and became something more the toy it started as. I was pretty arrogant in my opinions about the Rails community at first, but then I ended up working on several Rails projects over the years.
It still hides a lot under the hood, there are still arguably better technical frameworks out there and plenty of folks use it improperly, when they don't need to, and without really understanding the fundamentals, meaning that they tend to get in trouble when pushing the limits or moving outside the golden path of development.
And I feel the same way about k8s. I think it started out without anywhere near the features of similar frameworks. It didn't scale well, was simplistic in it's model, and overly complicated in it's implementation. But it was much more approachable than something like Mesos and answered the question of "why am I containerizing everything?", giving everything a purpose to those who started down the path of Docker. And now it has a huge following and industry behind it.
At this point, I've learned that what becomes popular isn't necessarily the "right" or "correct" architecture. There's a lot more to programming trends (and fads) than that. The whole industry seems to want to reinvent the wheel every decade, almost like some sort of planned obsolescence to justify our work. Nevertheless, it's rarely wise to fight against the tide and when you have the enough of the industry moving in a direction, we can make even toys into real tools.
- meatmanek 6y agoI actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go. Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscaling that is annoying to calculate in Mesos. (The Mesos master has no idea how many pods each framework _wants_ to launch, so we ended up writing code that knows how to deduce the resource demand from each framework's API or UI.) On the other side of the same coin, the framework in Mesos has no idea the total resources available to the cluster - it has to wait to be offered each box's resources in turn. This usually means frameworks either: a) accept the first offer which matches the basic requirements for a task, even if there are better places to run that task, or b) accept every single offer, under the assumption that they're the only framework on the cluster, and then implement their own scheduler on top of this pool of resources. (I call this the "monoframework" approach.) The former approach is how Mesos is supposed to work, but can lead to all sorts of bad outcomes, e.g. all the copies of a Marathon service running on the same box, or Spark having to use timeouts to know when it should give up waiting for offers if it doesn't get enough right away. The latter approach can lead to better placements, but defeats the purpose of the two-level scheduler, as no other frameworks can use the resources that aren't being used by the monoframework. Under Kubernetes, any operator can query the state of the cluster and make informed decisions about what to request. 1. https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...
- jaaron 6y agoMy frame of reference for the argument about Mesos vs k8s is from earlier in the timeline when the k8s was up and coming and just starting to compete with Mesos. Just as in my analogy with Rails: at it's inception, it lacked a bunch of features, but it was accessible. Years later, Rails had many of those features. k8s is the same. Operators didn't come around until late 2016 by CoreOS and even then they weren't widely adopted. It wasn't until after RedHat bought CoreOS and pushed operators as a pattern into k8s that the pattern took off. As it is, the k8s version of the operator framework only went 1.0 last year. Finally, I should have pointed it out in my original comment, the Mesos vs k8s comparison isn't perfect apples and apples. Mesos is just a component in a stack, whereas k8s is effectively a full stack. Again, Rails was pitched as an opinionated, batteries-included framework compared to many of the more focused frameworks that, for their niche, may have been better, but the convenience of having everything in one packages won out.