4 ms·

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google invested in it. Let's not forget John Wilkes (Google Borg, Omega, Kubernetes) on the stage at MesosCon in 2014 https://www.youtube.com/watch?v=VQAAkO5B5Hg https://www.youtube.com/watch?v=VQAAkO5B5Hg While I don't know the exact reasons behind Kubernetes becoming the scheduler and resource manager, I think that has very much to do with the stewardship of the Mesos project as well. By 2015/16 most Mesos committers were at Mesosphere. Suffice to say the open-governance was more of pain than a benefit to them. If you were at Apple, Adobe and others that relied on Mesos you didn't have much to say. Everything shifted towards DCOS and everyone was pushed towards commercial licenses. This is said, because sadly Kubernetes didn't fix the whole problem and left us with a distributed system with a text based "API" that requires text patching to manage and borderline clueless about the lifecycle of the services running on top. Yet, it's the best we got :)

jaaron 6y ago
This is much more the real story but I doubt most folks will ever hear it. Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently. Then again, k8s succeeded in part due to the Google reputation (even if undeserved) and it's use of GoLang. I always found Mesos' use of C++ meant many of the users (often developing in Java, Go, or Node) just wouldn't contribute.
- aabhay 6y ago
  I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.
  - jaaron 6y ago
    Eh, maybe. I see your point. The thing is, Mesos on it's own isn't k8s. Mesos is a scheduler. k8s is a full stack. So it's always been a bit off to fully compare the two. If your system fit clearly within what k8s did well, then you were fine and the stack worked. If you had a more complicated architecture with parts that didn't play well with containers or k8s' scheduling model, life became pretty difficult. That's one reason I liked Mesos is you could build the stack you needed for _your_ infrastructure. Granted, I don't think most shops had the kind of problems that warranted Mesos, let alone k8s (that's still true today). But k8s is "good enough" for lots of problems and understandably that's where the community went.
rad_gruchalski 6y ago
> Everything shifted towards DCOS and everyone was pushed towards commercial licenses. This was then partially diluted when Microsoft decided to deploy DC/OS and, from what I've heard, pushed Mesosphere to open source large chunks of DC/OS.
- dharmab 6y ago
  And then ACS was pretty much abandoned not long after that.
dharmab 6y ago
I'm an engineer at one of those companies you mentioned who worked directly on the Mesos+DC/OS platform. Mesosphere's support was pretty much nonexistent as far as we were concerned. We had major issues open for years without any significant action. They were never resolved. We had to solve most problems ourselves. We got so fed up that a few of us worked late nights (often up to 2AM) on a bottom-up skunkworks project to replace Mesos with Kubernetes. That project was exceptionally successful. (I'd like to point out that we are not in the Bay area and are adults with families and small children- we hated Mesos so much we were willing to stay up anyway.) In short, Mesos has a lot of great theory but we hated it in practice. Kubernetes has some theoretical flaws but it is better experience for practitioners.
justicezyx 6y ago
> Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it Brilliant on paper. The idea was influenced by MR workloads on Borg. Because MR is so large a workload pattern on Borg, that it benefits from having its own scheduler, and Borg also benefit from unnecessary meddling. The idea was only brilliant for a very small number of human users, both inside Google and in the broader industry. In terms of CPU & memory usage, mesos' approach should always take great share, because workloads that in the category of needing their own scheduler always is the big ones. But as an open source product, its adopters are far off from that group. So you can see that all early adopters of Mesos are large corps. I seldom hear any startup applaud Mesos. Precisely, I have repeated many times when Mesos was still relevant, that its model makes it costly to get started. It will never succeed unless it start to package a product that can provide the scheduler part. We all know what happened after K8s. K8s works because Borg already worked for 12+ years. > an elegant implemention that allowed for much more than what Kubernetes was desgined to. By design K8s is more capable then Mesos. Because K8s includes scheduler. And numerous other capability. Heck, excluding performance and scalability, K8s is a level a bove Borg in terms of feature and capability. Mesos is far behind Borg even, not mentioning K8s. Mesos does offer better scalability, on paper. I had no experience though. > Kubernetes was supposed to be scheduler for Mesos and Google invested in it. I never heard of such plan.
- jaaron 6y ago
  >> Kubernetes was supposed to be scheduler for Mesos and Google invested in it. > I never heard of such plan. Oh, there was a plan: https://github.com/kubernetes-retired/kube-mesos-framework https://github.com/kubernetes-retired/kube-mesos-framework But it wasn't really in anyone's business model interests, even if it may have made sense for customers and the community.
  - justicezyx 6y ago
    Sure, this type of early idea happens to any high-profile project. I meant to say that that I never heard any significant decision maker had that plan on K8s' future.
aabhay 6y ago
100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later) The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new cool easy-to-use distributed systems cropping up every week. The company even developed a sham 'community edition' that was designed to fail. One person complained on our community slack channel that he left his cluster running overnight and was charged $600 for the 12 hours of use. Within a few months of joining, I started to point out their odd technical decisions (e.g. why they decided to build their enterprise edition on Core OS rather than a trustworthy distro), and was eventually chopped for speaking up. I was fired right before a massive company offsite. When the rest of my team came back, they along with my manager were fired, too. The messsage was: we didn't need your whole team but because you spoke up we had to double screw you.
- dharmab 6y ago
  Ex-DC/OS user. This explains a lot. One thing I'd like to say is that CoreOS was the best component of DC/OS by a long mile. We still use its fork (Flatcar) today.
  - aabhay 6y ago
    That’s cool! The issue is that when you’re selling to a massive legacy institution, the infosec on a new Linux distro that self updates to new patches is really no bueno. We lost many months on this piece and on key deals. I heard from the grapevine that they ended up having to port DCOS to Ubuntu after all, soon after I left.
    - dharmab 6y ago
      Yeah we disabled the auto updates and orchestrated the updates ourselves. Saved our bacon multiple times when kernel/systemd/docker bugs got pushed out.
- deleted 6y ago

jaaron 6y ago

This is much more the real story but I doubt most folks will ever hear it. Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently. Then again, k8s succeeded in part due to the Google reputation (even if undeserved) and it's use of GoLang. I always found Mesos' use of C++ meant many of the users (often developing in Java, Go, or Node) just wouldn't contribute.

aabhay 6y ago
I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.
- jaaron 6y ago
  Eh, maybe. I see your point. The thing is, Mesos on it's own isn't k8s. Mesos is a scheduler. k8s is a full stack. So it's always been a bit off to fully compare the two. If your system fit clearly within what k8s did well, then you were fine and the stack worked. If you had a more complicated architecture with parts that didn't play well with containers or k8s' scheduling model, life became pretty difficult. That's one reason I liked Mesos is you could build the stack you needed for _your_ infrastructure. Granted, I don't think most shops had the kind of problems that warranted Mesos, let alone k8s (that's still true today). But k8s is "good enough" for lots of problems and understandably that's where the community went.

rad_gruchalski 6y ago

> Everything shifted towards DCOS and everyone was pushed towards commercial licenses. This was then partially diluted when Microsoft decided to deploy DC/OS and, from what I've heard, pushed Mesosphere to open source large chunks of DC/OS.

dharmab 6y ago
And then ACS was pretty much abandoned not long after that.

dharmab 6y ago

I'm an engineer at one of those companies you mentioned who worked directly on the Mesos+DC/OS platform. Mesosphere's support was pretty much nonexistent as far as we were concerned. We had major issues open for years without any significant action. They were never resolved. We had to solve most problems ourselves. We got so fed up that a few of us worked late nights (often up to 2AM) on a bottom-up skunkworks project to replace Mesos with Kubernetes. That project was exceptionally successful. (I'd like to point out that we are not in the Bay area and are adults with families and small children- we hated Mesos so much we were willing to stay up anyway.) In short, Mesos has a lot of great theory but we hated it in practice. Kubernetes has some theoretical flaws but it is better experience for practitioners.

justicezyx 6y ago

> Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it Brilliant on paper. The idea was influenced by MR workloads on Borg. Because MR is so large a workload pattern on Borg, that it benefits from having its own scheduler, and Borg also benefit from unnecessary meddling. The idea was only brilliant for a very small number of human users, both inside Google and in the broader industry. In terms of CPU & memory usage, mesos' approach should always take great share, because workloads that in the category of needing their own scheduler always is the big ones. But as an open source product, its adopters are far off from that group. So you can see that all early adopters of Mesos are large corps. I seldom hear any startup applaud Mesos. Precisely, I have repeated many times when Mesos was still relevant, that its model makes it costly to get started. It will never succeed unless it start to package a product that can provide the scheduler part. We all know what happened after K8s. K8s works because Borg already worked for 12+ years. > an elegant implemention that allowed for much more than what Kubernetes was desgined to. By design K8s is more capable then Mesos. Because K8s includes scheduler. And numerous other capability. Heck, excluding performance and scalability, K8s is a level a bove Borg in terms of feature and capability. Mesos is far behind Borg even, not mentioning K8s. Mesos does offer better scalability, on paper. I had no experience though. > Kubernetes was supposed to be scheduler for Mesos and Google invested in it. I never heard of such plan.

jaaron 6y ago
>> Kubernetes was supposed to be scheduler for Mesos and Google invested in it. > I never heard of such plan. Oh, there was a plan: https://github.com/kubernetes-retired/kube-mesos-framework https://github.com/kubernetes-retired/kube-mesos-framework But it wasn't really in anyone's business model interests, even if it may have made sense for customers and the community.
- justicezyx 6y ago
  Sure, this type of early idea happens to any high-profile project. I meant to say that that I never heard any significant decision maker had that plan on K8s' future.

aabhay 6y ago

100% Agreed. Ex-Mesosphere employee (I joined in 2015 and left/was fired a year later) The dominant ethos at Mesosphere was that they already won, and were poised to become the next 'cloud orchestration' above the cloud services. But the managers also had no empathy for developer experience -- the majority opinion was "distributed systems are hard, developers don't deserve to have a good experience", despite the new cool easy-to-use distributed systems cropping up every week. The company even developed a sham 'community edition' that was designed to fail. One person complained on our community slack channel that he left his cluster running overnight and was charged $600 for the 12 hours of use. Within a few months of joining, I started to point out their odd technical decisions (e.g. why they decided to build their enterprise edition on Core OS rather than a trustworthy distro), and was eventually chopped for speaking up. I was fired right before a massive company offsite. When the rest of my team came back, they along with my manager were fired, too. The messsage was: we didn't need your whole team but because you spoke up we had to double screw you.

dharmab 6y ago
Ex-DC/OS user. This explains a lot. One thing I'd like to say is that CoreOS was the best component of DC/OS by a long mile. We still use its fork (Flatcar) today.
- aabhay 6y ago
  That’s cool! The issue is that when you’re selling to a massive legacy institution, the infosec on a new Linux distro that self updates to new patches is really no bueno. We lost many months on this piece and on key deals. I heard from the grapevine that they ended up having to port DCOS to Ubuntu after all, soon after I left.
  - dharmab 6y ago
    Yeah we disabled the auto updates and orchestrated the updates ourselves. Saved our bacon multiple times when kernel/systemd/docker bugs got pushed out.
deleted 6y ago