4 ms·

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some exte

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the competition. Mesos was trying to become a solution seemingly from scratch. I worked with and interacted with a number of Mesos project members and committers, and while they were generally bright folks, their industry experience was surprisingly shallow. In my experience, Mesos changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile. Within a year Kubernetes was waaaaay ahead and Mesos started considering pivoting to support k8s. I no longer see a purpose in Mesos, and frankly, that's okay. Too many Apache projects are on eternal life support, and they lower the bar for the rest of the Apache ecosystem, which has sort of begun to earn a poor reputation for quality (ala some of the fringe Boost libraries). Apache Foundation is no longer a label for solid, reliable software.

dharmab 6y ago
We used Mesos in production until 2020 (started transition to Kubernetes in 2018), and this comment is incredibly accurate. Mesos was an interesting project but the defaults were incredibly naive about production environments. Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'.
- bogomipz 6y ago
  >"Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'." What was Mesos default behavior when a node was powered off?
  - dharmab 6y ago
    It assumed that the Node had a temporary network fault and did not automatically remove the Node. Marathon also did not automatically schedule replacement containers on other Nodes. Kubernetes will, after a period of time, garbage collect the Node and schedule replacements as dictated by the controller.
hintymad 6y ago
A challenge with Mesos is that Mesos was a piece of technology, a framework at most, instead of a product. When I was using Mesos, the selling point was flexible and efficient resource scheduling. Unfortunately, resource/machine efficiency alone does not sell well, as most of the companies and individuals have betters things to worry about, say, productivity.
- ghaff 6y ago
  >the selling point was flexible and efficient resource scheduling There was a period when it wasn't clear that you didn't need both resource management and container orchestration. One of my colleagues was quite convinced at the time that we needed both Mesos and Kubernetes. If course, the market coalesced around Kubernetes which largely backfilled the missing capabilities.
  - brutus1213 6y ago
    Curious how you define resource manager. Is that the scheduler or something else?
- ignoramous 6y ago
  > Unfortunately, resource/machine efficiency alone does not sell well... Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building and running a data center (which is to say, folks pay through their nose for servers yet those are under-utilized by a huge margin) [0] [0] https://youtu.be/dInADzgCI-s?t=535 https://youtu.be/dInADzgCI-s?t=535
  - hintymad 6y ago
    I don't deny that. It's just that so many companies have so much inefficiency else where that addressing resource inefficiency has too low a marginal return.
nemothekid 6y ago
>As neat as Mesos was, it was doomed from the start. I don't think it was doomed from the start; Mesos was deployed at Twitter 4 years before k8s was even released. The "problem" with Mesos was that it was focused on a future that never panned out. The "killer app" for Mesos was Spark, and the problem they were focused on solving was resource allocation for batch jobs. Mesos was supposed to be a replacement for YARN. Even Marathon, which was the de facto application launcher, was pitched as a "meta-framework". Marathon wasn't for launching services, it was for launching custom frameworks. They never really pivoted from this, and the writing was on the wall when Mesosphere decided to make Marathon as proprietary as possible and fully integrate it within their commercial solution. Once Marathon was gone, Mesos didn't really compete with k8s anymore unless you wanted to write your own framework.
- xyzzy_plugh 6y ago
  You're partially correct. Mesos was originally an academic project out of UC Berkeley. I'm not aware of what industry connections there were, but the story is not at all similar to Kubernetes and Borg. Aurora was Twitter's contribution -- a bit missing piece of Mesos at the time. It definitely steered Mesos towards solving for Spark, but I'm really not sure what Mesos was actively solving before then.
- tknaup 6y ago
  Co-founder of Mesosphere/D2iQ here. Mesos started as a research project at Berkeley in 2009 and was originally focused on cluster computing frameworks like Hadoop. From the paper: "We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI." It actually predates YARN by a few years. But, it very quickly (in 2010) saw production use at Twitter as the foundation for Twitter's custom PaaS which was later open sourced as Apache Aurora. Marathon's main use case was actually for running microservice application in containers, which is why it has some advanced features around managing groups of containerized apps and their dependencies. The "meta-framework" use case for launching custom frameworks was also important but basically just needs Marathon to keep a container alive. Mesosphere never made Marathon proprietary. The full code is still OSS here: https://github.com/mesosphere/marathon/ https://github.com/mesosphere/marathon/ Our commercial product DC/OS just added advanced workflows through a UI on top, and better integration with the rest of the components around Mesos.
tannhaeuser 6y ago
> Apache Foundation is no longer a label for solid, reliable software ASF is just a shelter for providing governance, infra, a legal framework for donations, and credibility to projects that went unprofitable, or weren't intended to be profitable to begin with, such as complements to commercial software. It's a responsible way to dump software you can't realistically maintain going forward, to give it a chance to build up a community by those depending on it, and often accompanied by an initial contribution to ASF. While not every software project is doing well for sure, whether ASF or not, I think the above sentence really misrepresents what ASF is doing (and arguably is doing with a lot of success).

dharmab 6y ago

We used Mesos in production until 2020 (started transition to Kubernetes in 2018), and this comment is incredibly accurate. Mesos was an interesting project but the defaults were incredibly naive about production environments. Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'.

bogomipz 6y ago
>"Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'." What was Mesos default behavior when a node was powered off?
- dharmab 6y ago
  It assumed that the Node had a temporary network fault and did not automatically remove the Node. Marathon also did not automatically schedule replacement containers on other Nodes. Kubernetes will, after a period of time, garbage collect the Node and schedule replacements as dictated by the controller.

hintymad 6y ago

A challenge with Mesos is that Mesos was a piece of technology, a framework at most, instead of a product. When I was using Mesos, the selling point was flexible and efficient resource scheduling. Unfortunately, resource/machine efficiency alone does not sell well, as most of the companies and individuals have betters things to worry about, say, productivity.

ghaff 6y ago
>the selling point was flexible and efficient resource scheduling There was a period when it wasn't clear that you didn't need both resource management and container orchestration. One of my colleagues was quite convinced at the time that we needed both Mesos and Kubernetes. If course, the market coalesced around Kubernetes which largely backfilled the missing capabilities.
- brutus1213 6y ago
  Curious how you define resource manager. Is that the scheduler or something else?
ignoramous 6y ago
> Unfortunately, resource/machine efficiency alone does not sell well... Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building and running a data center (which is to say, folks pay through their nose for servers yet those are under-utilized by a huge margin) [0] [0] https://youtu.be/dInADzgCI-s?t=535 https://youtu.be/dInADzgCI-s?t=535
- hintymad 6y ago
  I don't deny that. It's just that so many companies have so much inefficiency else where that addressing resource inefficiency has too low a marginal return.

nemothekid 6y ago

>As neat as Mesos was, it was doomed from the start. I don't think it was doomed from the start; Mesos was deployed at Twitter 4 years before k8s was even released. The "problem" with Mesos was that it was focused on a future that never panned out. The "killer app" for Mesos was Spark, and the problem they were focused on solving was resource allocation for batch jobs. Mesos was supposed to be a replacement for YARN. Even Marathon, which was the de facto application launcher, was pitched as a "meta-framework". Marathon wasn't for launching services, it was for launching custom frameworks. They never really pivoted from this, and the writing was on the wall when Mesosphere decided to make Marathon as proprietary as possible and fully integrate it within their commercial solution. Once Marathon was gone, Mesos didn't really compete with k8s anymore unless you wanted to write your own framework.

xyzzy_plugh 6y ago
You're partially correct. Mesos was originally an academic project out of UC Berkeley. I'm not aware of what industry connections there were, but the story is not at all similar to Kubernetes and Borg. Aurora was Twitter's contribution -- a bit missing piece of Mesos at the time. It definitely steered Mesos towards solving for Spark, but I'm really not sure what Mesos was actively solving before then.
tknaup 6y ago
Co-founder of Mesosphere/D2iQ here. Mesos started as a research project at Berkeley in 2009 and was originally focused on cluster computing frameworks like Hadoop. From the paper: "We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI." It actually predates YARN by a few years. But, it very quickly (in 2010) saw production use at Twitter as the foundation for Twitter's custom PaaS which was later open sourced as Apache Aurora. Marathon's main use case was actually for running microservice application in containers, which is why it has some advanced features around managing groups of containerized apps and their dependencies. The "meta-framework" use case for launching custom frameworks was also important but basically just needs Marathon to keep a container alive. Mesosphere never made Marathon proprietary. The full code is still OSS here: https://github.com/mesosphere/marathon/ https://github.com/mesosphere/marathon/ Our commercial product DC/OS just added advanced workflows through a UI on top, and better integration with the rest of the components around Mesos.

tannhaeuser 6y ago

> Apache Foundation is no longer a label for solid, reliable software ASF is just a shelter for providing governance, infra, a legal framework for donations, and credibility to projects that went unprofitable, or weren't intended to be profitable to begin with, such as complements to commercial software. It's a responsible way to dump software you can't realistically maintain going forward, to give it a chance to build up a community by those depending on it, and often accompanied by an initial contribution to ASF. While not every software project is doing well for sure, whether ASF or not, I think the above sentence really misrepresents what ASF is doing (and arguably is doing with a lot of success).