14 ms·

Apache Mesos to be moved to Attic

249 points by coolandsmartrr 6y ago

fiberoptick 6y ago
End of an era... Are there any viable alternatives to Kubernetes and Nomad?
- dragonsh 6y ago
  For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain. [1] https://www.linuxcontainers.org/ https://www.linuxcontainers.org/
  - marcinzm 6y ago
    >except for managed services from cloud providers I'd imagine most small to medium companies would be running things on a cloud service using managed kubernetes. It seems mostly larger companies that are sticking with non-cloud hardware and services. The advantage of kubernetes in that case is that there is a large ecosystem of helm charts, guides, documentation, etc. Deploying something new from scratch is fairly easy since someone else has done all the leg work and packaged it all up.
    - rantwasp 6y ago
      just because something is packaged does not mean it’s usable. YMMV but this is how security horror stories start. Someone ran a container they had no idea where it came from, happily used a helm chart. Most of the times it’s not even malicious - it’s outdated software because “it just works”
      - marcinzm 6y ago
        In my experience all the common helm charts and docker images are regularly updated. If you don't update your installation of them then you also wouldn't update a docker compose or LXD.
        rantwasp 6y ago
        that’s a big statement. what constitutes “common” helm charts?
  - mplewis 6y ago
    For a different perspective, learning Kubernetes and using it widely gives you a universal set of tools for managing all sorts of applications at all scales. https://news.ycombinator.com/item?id=26502900 https://news.ycombinator.com/item?id=26502900
    - ForHackernews 6y ago
      Sure, this is kind of true, but only in the most depressing way possible. Kubernetes is overengineered and terrible but it's also just about the only game in town if you want a vendor-agnostic container orchestration system. This is the same situation as Javascript circa 2008: "Learning this absolute dogshit language and using it widely will give you the ability to write universal applications that will run in browsers everywhere and make you very employable." You're not wrong about k8s today, and wouldn't have been wrong about JS in the past, but boy is it a sad indictment of our industry that these things are true.
      - marcinzm 6y ago
        Kubernetes is engineered to solve the five hundred different problems that different people have in a way that works for them. If you don't do that then everyone will complain about their feature or some edge case being missing and not use the system (see comments on mesos in this thread). That's not over engineering, that's the required amount of engineering for this sort of system.
        ForHackernews 6y ago
        Yeah, k8s people keep telling me this. But also, k8s "secrets" are not, in fact, secret, you can't actually accept traffic from the web without some extra magic cloud load balancer (cf https://v0-3-0--metallb.netlify.app/ https://v0-3-0--metallb.netlify.app/ maybe eventually) or properly run anything stateful like a database (maybe soon). Forget covering "everyone's" use cases: From where I'm sitting, k8s is an insanely complicated system that does a miserable job of covering the 95% use cases that Heroku solved in 2008. It's great that k8s (maybe) solves hard problems that nobody except Google has, but it doesn't solve the easy problems that most people have.
  - 0xEFF 6y ago
    > For over 90% of workloads kubernetes is an overkill. It's not. Take any simple web app and deploy it into managed GKE with Anthos and you automatically get SLI's and the ability to define SLO's for availability and latency with a nice wizard. Takes a few minutes to setup. The amount of engineering needed to achieve good SLO monitoring dwarfs the engineering needed to run a simple app so it just never happened. That's no longer the case. > Only when company is reaching google scale kubernetes make sense. Also obviously not true given the number of companies deriving value from kubernetes. Note, I'm a Google Cloud partner.
    - dragonsh 6y ago
      Your statement already support that without the blessings of engineering team of Amazon, Google, Microsoft, Digital Ocean and various managed kubernetes service it's impossible for a reasonable small team to manage and monitor k8s and all of this service comes with lock-in and additional capital outlay. Obviously for a Google Cloud Partner, more people are tied to gcp and kubernetes, higher the revenue. Its secondary if it's really necessary for an application to require k8s.
      - deleted 6y ago
        [deleted]
  - orthecreedence 6y ago
    I don't know about this. I caution people against microservices architecture all the time, but at my company having a scheduler just made sense. We spin up and down queue workers by massive amounts every hour, and doing this with anything besides a scheduler would be really tricky. Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.
    - p_l 6y ago
      It doesn't even have to be some microservices monster, just trying to bin-pack a bunch of different services, even monoliths, onto a fleet of servers. If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)
  - lima 6y ago
    > For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. I hear people repeating this truism all day, but from practical experience, it doesn't seem to be the case - Kubernetes is paying dividends even in small single-node setups.
    - lovedswain 6y ago
      K8s single noder and GKE user here, can confirm I wouldn't remotely even consider going back. Deploying an app takes 5-10 minutes at most first time around, new pushes <1 minute, and when there is ops bullshit involved, it is never wasted work, and never needs to be repeated twice. I hated Kubernetes ops complexity at first, but there really isn't that much to it, and if it's too much, a service like GKE takes 70%+ of it away from you
  - madhadron 6y ago
    > Only when company is reaching google scale kubernetes make sense. Long before Google scale. Kubernetes won't handle clusters anywhere near that large.
- desktopninja 6y ago
  Personally think AWS ECS is a 3rd place contender. Would add sprinkles to it if only they'd allow yaml files vs json configs in the aws-cli. ecs-cli and copilot are alright. Generally prefer to stay as close as possible to aws-cli
  - kbar13 6y ago
    AWS CDK makes ECS a lot more palatable
  - whatsmyusername 6y ago
    In AWS I’d choose ECS over any k8s variation on price alone before you get to the IAM integrations.
- fastball 6y ago
  Docker-swarm?
  - mplewis 6y ago
    Docker Swarm is being cannibalized by k8s.
  - yjftsjthsd-h 6y ago
    Having worked at a company running Docker swarm at... medium(?) scale... I have witnessed a truly shocking variety of bugs in its network stack. I always wondered if it was something specific to the company setup, but the result is that we just couldn't stay on a platform that would randomly fail to give containers a working network connection to each other.
- ForHackernews 6y ago
  Flockport? https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai...
zekrioca 6y ago
I'm sad for that.. It was truly a neat project with many great ideas, but Kubernetes influence brought it to an end :/
specialp 6y ago
We used Mesos as our first container orchestration stack. It worked OK but Kubernetes came along and offered a one stop solution for everything. In Mesos you had to use separate projects for container management (Marathon), discoverability (Consul/HAPROXY). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has. This was in 2014 so it was early but k8s came along with everything we needed.
kevincox 6y ago
I used mesos a while ago across a couple of cheep VPS for personal projects and services and it was great. I especially liked that I could used a shell executor to run services with filesystem access. For many environments this probably didn't make sense but for me I could put some secrets on the filesystem and services could make unix sockets available to nginx for SSL and hostname based forwarding. Also with Nix I could easily install software the specific services needed and it was trivial to integrate into the garbage collector making it much faster than launching a docker container. That being said the project wasn't moving that fast and it was a bit buggly for me and nodes would regularly get into loops where they couldn't authenticate with the master for various reasons (I think there were timing issues and timeouts that caused issues over the internet). Now I'm using Kubernetes but I have all of this complicated virtual networking that I don't need, I'm locked into docker-like containers and the thing is so complicated that I need to pay someone to run it for me.
xyzzy_plugh 6y ago
I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the competition. Mesos was trying to become a solution seemingly from scratch. I worked with and interacted with a number of Mesos project members and committers, and while they were generally bright folks, their industry experience was surprisingly shallow. In my experience, Mesos changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile. Within a year Kubernetes was waaaaay ahead and Mesos started considering pivoting to support k8s. I no longer see a purpose in Mesos, and frankly, that's okay. Too many Apache projects are on eternal life support, and they lower the bar for the rest of the Apache ecosystem, which has sort of begun to earn a poor reputation for quality (ala some of the fringe Boost libraries). Apache Foundation is no longer a label for solid, reliable software.
- dharmab 6y ago
  We used Mesos in production until 2020 (started transition to Kubernetes in 2018), and this comment is incredibly accurate. Mesos was an interesting project but the defaults were incredibly naive about production environments. Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'.
  - bogomipz 6y ago
    >"Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'." What was Mesos default behavior when a node was powered off?
    - dharmab 6y ago
      It assumed that the Node had a temporary network fault and did not automatically remove the Node. Marathon also did not automatically schedule replacement containers on other Nodes. Kubernetes will, after a period of time, garbage collect the Node and schedule replacements as dictated by the controller.
johncena33 6y ago
How does this impact the company behind Mesos and products based on Mesos, e.g., Mesosphere etc.?
- avita1 6y ago
  Mesospehere was renamed and has shifted to supporting Kubernetes. https://d2iq.com/blog/mesosphere-is-now-d2iq https://d2iq.com/blog/mesosphere-is-now-d2iq
- rad_gruchalski 6y ago
  Mesosphere is now D2iQ, they have their own Kubernetes and stopped supporting Mesos / Marathon in October 2020.
kraemate 6y ago
Mesos made a ton of new contributions to distributed resource management. Resource offers and application integration can allow high cluster utilization and efficiency. Giving applications the opportunity to take part in resource allocation and scheduling was similar to the Exokernel design, and led to many interesting middleware architectures. Mesos also introduced Dominant Resource Fairness for allocating resources to competing applications, which has become an important concept in CS and specifically computational economics in its own right. Mesos achieved the rare combination of using CS research towards a widely deployed operational system. Its C++ code is simple enough to understand and hack, and seemed like one of those projects that you can/want to reimplement on your own "in a few days". It is sad how quickly it has become obsolete.
- jeffbee 6y ago
  Are there examples of high-utilization, large-scale Mesos deployments? Mesos didn't even gain over-commit until 2015, so it seems like it was generally behind the state of the art.
  - dharmab 6y ago
    Most famously, Siri (used to?) run on a very large scale Mesos deployment (10000s of nodes, much higher than Kubernetes can scale to). Unfortunately the original article is lost, but here's a summary: https://daringfireball.net/linked/2015/04/29/siri-apache-mesos https://daringfireball.net/linked/2015/04/29/siri-apache-mes...
    - jeffbee 6y ago
      OK but what was the utilization? I'm not really sure K8s is state-of-the-art either. There are published research papers about very-large-scale clusters with 80%+ resource utilization.
      - dharmab 6y ago
        In our production experience, utilization had far more to do with the service owners (or autoscalers/auto-tuners) correctly choosing the cgroups and CPU scheduler allocations, as well as the kernel settings for cgroup slicing and CPU scheduler. We had Mesos clusters with 3% utilization and have Kubernetes clusters with 95%+ utilization. But we also have Kubernetes clusters with <10% utilization.
fnord123 6y ago
Fun fact: According to the initial Mesos paper: Spark was a demo project to show off Mesos. https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf > To validate our hypothesis that specialized frameworks providevalue over general ones, we have also built a new frame-work on top of Mesos called Spark ...
- zekrioca 6y ago
  Look at the authors list. Some of them also participated in creating and designing Spark.
  - fnord123 6y ago
    Yes, this is not a surprise. The fun fact part is that Spark began as merely a way to validate the hypothesis of Mesos and look how the hypothesis validation project has taken off. Neat!
yuppie_scum 6y ago
RIP. It was a strong contender for a while before Kube matured.
jeffbee 6y ago
I wonder what recent large-scale adopters of Mesos are going to do. I know Dropbox deployed Mesos in the last year or so[1], even though it was pretty obvious that the project community was dead. Will the existing users found a new community? https://www.amd.com/system/files/documents/new-epyc-case-study-dropbox.pdf https://www.amd.com/system/files/documents/new-epyc-case-stu...
benjaminwootton 6y ago
I went to see Mesos early in their life in their San Francisco office after a joint customer put us in touch. Never in my life did I meet such an arrogant group of people. First, they left us waiting in reception for an hour. They eventually took the meeting over lunch, where we had to watch them inhaling their free food. Some guy in plastic-leather trousers spent most of the hour lecturing us about all of the multi million deals they were doing, and how they weren't interested in speaking with anyone who couldn't write 7 figure checks. Not once did they ask about our business or even our names if I remember correctly. I had a similar experience with other west coast tech companies. Too arrogant for their own good and not able to put in the hard yards with stodgy old enterprise companies even when they have good technology and are early to market.
- eklitzke 6y ago
  You're thinking of Mesosphere, a separate company building a product on top of Mesos.
  - benjaminwootton 6y ago
    True. As the major corporate sponsor though they didn’t give it the best chance. I ran one of the first Docker partners and saw first hand how both Mesos and Docker Inc had considerable mindset and interest from large enterprises for a year or so before Kubernetes matured. They both spectacularly failed to deliver through arrogance and bad execution.
    - pm90 6y ago
      I remember visiting SFBA in 2014 or so, a friend worked in PayPal said they used mesos, kubernetes was a baby project with a lot of hype. It sure seemed mesos had won back then, and the fact that kubernetes caught up, surpassed and then succeeded so wildly would have never crossed my mind.
      - smarterclayton 6y ago
        I worked with tech companies and some banks in the early days of Kubernetes who were deep in Mesos and many of them felt the same way. It’s really hard to see this as it’s happening. Kubernetes is absolutely “worse is better” when it comes to scheduling. But scheduling isn’t the problem most people needed solved. Kubernetes standardized deployment patterns of multiple apps well for most people. Everything else was a nice to have (scheduling, scale, ease of initial setup, etc).
ksec 6y ago
I remember at MesosCon, Apple said they are the largest Mesos user. I wonder if they also moved to K8s as well.
- thor24 6y ago
  They have apparently hired a ton of great k8 folks who have been tasked with building an internal layer of sorts for themselves so I guess they are moving off it (though not anytime soon).
- Conan_Kudo 6y ago
  Apple was at KubeCon last year, so I assume they've moved to Kubernetes too.
fh973 6y ago
The framework-based scheduler architecture looked like an interesting concept at first sight. But its advantage (write your custom scheduling policy, along with gaining scalability) evidently wasn't that important to make up for the effort of actually having to write your own framework to run anything. (In this regard its a bit like object storage as a file system replacement narrative, which promises scalability, if you adapt your application). This seems to be true both for the Omega experiments at Google (which I assume still uses Borg), and for Mesos. In the end, Mesos always lacked even a minimal application deployment story. Marathon unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API. I think if Mesos(-sphere) had recognized that gap in time, and came up with a decent framework for application deployments, instead of telling everyone to write their own framework (in C++, against a hacky code base), Kubernetes might not have had a chance, or at least we would have two alternatives to chose from. Too bad.
jaaron 6y ago
I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning: k8s is the Rails of the cloud. Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterprise web frameworks, Rails felt like a toy. It didn't have the features, robustness, safety, and scalability of "proper" frameworks. What did Rails do? It was easy to get started and it hid a lot of the boring and painful work of web frameworks at the time. Through the sheer force of will of a massive community, Rails grew up and became something more the toy it started as. I was pretty arrogant in my opinions about the Rails community at first, but then I ended up working on several Rails projects over the years. It still hides a lot under the hood, there are still arguably better technical frameworks out there and plenty of folks use it improperly, when they don't need to, and without really understanding the fundamentals, meaning that they tend to get in trouble when pushing the limits or moving outside the golden path of development. And I feel the same way about k8s. I think it started out without anywhere near the features of similar frameworks. It didn't scale well, was simplistic in it's model, and overly complicated in it's implementation. But it was much more approachable than something like Mesos and answered the question of "why am I containerizing everything?", giving everything a purpose to those who started down the path of Docker. And now it has a huge following and industry behind it. At this point, I've learned that what becomes popular isn't necessarily the "right" or "correct" architecture. There's a lot more to programming trends (and fads) than that. The whole industry seems to want to reinvent the wheel every decade, almost like some sort of planned obsolescence to justify our work. Nevertheless, it's rarely wise to fight against the tide and when you have the enough of the industry moving in a direction, we can make even toys into real tools.
- meatmanek 6y ago
  I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go. Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscaling that is annoying to calculate in Mesos. (The Mesos master has no idea how many pods each framework _wants_ to launch, so we ended up writing code that knows how to deduce the resource demand from each framework's API or UI.) On the other side of the same coin, the framework in Mesos has no idea the total resources available to the cluster - it has to wait to be offered each box's resources in turn. This usually means frameworks either: a) accept the first offer which matches the basic requirements for a task, even if there are better places to run that task, or b) accept every single offer, under the assumption that they're the only framework on the cluster, and then implement their own scheduler on top of this pool of resources. (I call this the "monoframework" approach.) The former approach is how Mesos is supposed to work, but can lead to all sorts of bad outcomes, e.g. all the copies of a Marathon service running on the same box, or Spark having to use timeouts to know when it should give up waiting for offers if it doesn't get enough right away. The latter approach can lead to better placements, but defeats the purpose of the two-level scheduler, as no other frameworks can use the resources that aren't being used by the monoframework. Under Kubernetes, any operator can query the state of the cluster and make informed decisions about what to request. 1. https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...
throwaway823882 6y ago
Most of the comments here about Mesos are exactly my experience with Kubernetes. Replacing the words and I'm just nodding my head: "Kubernetes changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile." "Kubernetes was an interesting project but the defaults were incredibly naive about production environments." "Kubernetes unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API." "A challenge with Kubernetes is that Kubernetes was a piece of technology, a framework at most, instead of a product." "In Kubernetes you had to use separate projects for container management (kubelet, kube-scheduler, kube-controller-manager, kube-proxy, cri-o), discoverability (etcd). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has."
- dharmab 6y ago
  As someone you quoted: Take every problem you have with Kubernetes, multiply by 10x to 100x, and you have DC/OS. It's all perspective :)
random3 6y ago
It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google invested in it. Let's not forget John Wilkes (Google Borg, Omega, Kubernetes) on the stage at MesosCon in 2014 https://www.youtube.com/watch?v=VQAAkO5B5Hg https://www.youtube.com/watch?v=VQAAkO5B5Hg While I don't know the exact reasons behind Kubernetes becoming the scheduler and resource manager, I think that has very much to do with the stewardship of the Mesos project as well. By 2015/16 most Mesos committers were at Mesosphere. Suffice to say the open-governance was more of pain than a benefit to them. If you were at Apple, Adobe and others that relied on Mesos you didn't have much to say. Everything shifted towards DCOS and everyone was pushed towards commercial licenses. This is said, because sadly Kubernetes didn't fix the whole problem and left us with a distributed system with a text based "API" that requires text patching to manage and borderline clueless about the lifecycle of the services running on top. Yet, it's the best we got :)
- jaaron 6y ago
  This is much more the real story but I doubt most folks will ever hear it. Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently. Then again, k8s succeeded in part due to the Google reputation (even if undeserved) and it's use of GoLang. I always found Mesos' use of C++ meant many of the users (often developing in Java, Go, or Node) just wouldn't contribute.
  - aabhay 6y ago
    I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.
relistan 6y ago
The real gem of the Mesos ecosystem was the much lesser known Singularity scheduler from HubSpot. https://github.com/HubSpot/Singularity https://github.com/HubSpot/Singularity I've run this at scale, in production since 2015 and it has been absolutely rock solid and does most of the production things you'd want. Unlike commercial products, it was written to run HubSpot's own infrastructure so it does what a production system needs. Really bummed to have to downgrade to something like K8s in the near future.
- tpetr 6y ago
  Thanks for the kind words! Singularity is still a core piece of infrastructure for running stateless apps at HubSpot (our status page currently reads ~13k Requests, ~22k Active Tasks, ~700 Agents). We're also heavily investing in Kubernetes, but Singularity is solid enough that our focus has been more towards reliably running and scaling stateful things like MySQL, ZK, Kafka, HBase, and ES.
- chucky_z 6y ago
  There’s a Singularity driver for Nomad https://www.nomadproject.io/docs/drivers/external/singularity https://www.nomadproject.io/docs/drivers/external/singularit... Nomad is pretty darn easy to run and scales far. I help run about 20 clusters in day job from 100 to 1800 nodes, we’re going to push one of them to over 8500 nodes here soon too. We don’t use the Singularity driver but we wrote two of our own and they work well and are easy to maintain.
aggress 6y ago
Disclaimer - I used to work for Mesosphere. On the topic of arrogance, I wasn't in that meeting referenced, but having worked with most people there who could have been, and being a Brit who's worked for west coast startups like that for nearly 10 years, it's that American confidence, outlook and drive (along with Sand Hill road) that contributes to so much success out there. I'm a fan, but have to check myself every time i use 'super' as an adjective. Tech vs Product is a good comparison to make. I banged my head many times on DC/OS (the commercial version of Mesos) and underlying Mesos itself. Solid tech, but I'd have liked to have seen the product develop faster, to realise its potential in the enterprise market, which was huge.
benjamin_mahler 6y ago
I’m one of the long term PMC / committers on mesos. In retrospect I feel this was inevitable to a few key reasons: * k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster management systems. We learned many things along the way that we would do differently a second time around. * Mesos was too “batteries not included”: this fragmented the community, made it unapproachable to new users, and led to a lot of redundant effort. Most users just want to run services, jobs and cron jobs, but this was not part of Mesos and you had to choose from the array of ecosystem schedulers (e.g. Aurora, Marathon, Chronos, Singularity, etc) or building something yourself. * Mesosphere was a VC backed startup and drove the project after Twitter. Being a VC backed startup you need to have a business model that can generate revenue. This led to a lot of tensions and mistrust with users and other vendors. Compare this with Google / k8s, where Google does not need to generate revenue from k8s directly, it can instead invest massive amounts of money and strong engineers on the notion that it will improve Google’s cloud computing business. Even if k8s hadn’t come along, Mesos was ripe to be disrupted by something that was easier to use out of the box, and that had a benevolent home that could unite vendors and other contributors. Mesos could have perhaps evolved in this direction over time but we'll never know now.
- rad_gruchalski 6y ago
  Thanks for all the work!
- qbasic_forever 6y ago
  It's refreshing to see an honest and critical evaluation of things. This kind of decision should be celebrated and encouraged with all projects.
  - waynesonfire 6y ago
    On the announcement that it's getting shelved you get a retro. How is that refreshing? Whats your baseline? Where is D2iQ response to this news?
ta20210405 6y ago
Somewhat of an end of an era. Worked at a shop that had one of the largest Mesos fleets in the world. Thousands of physical nodes, petabytes of data. True Mesos in production was something to witness.
lars_francke 6y ago
Nothing is decided yet. This is a vote thread which _might_ end with someone interested in Mesos stepping in to pick the project up. This hasn't happened in a while with other projects (a whole bunch of Apache projects have just retired in the last few months) but with Mesos there might be a chance.
throwaway_1237 6y ago
What is the attic?
arnon 6y ago
The vote has been cancelled. New discussion - https://news.ycombinator.com/item?id=26748191 https://news.ycombinator.com/item?id=26748191

fiberoptick 6y ago

End of an era... Are there any viable alternatives to Kubernetes and Nomad?

dragonsh 6y ago
For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain. [1] https://www.linuxcontainers.org/ https://www.linuxcontainers.org/
- marcinzm 6y ago
  >except for managed services from cloud providers I'd imagine most small to medium companies would be running things on a cloud service using managed kubernetes. It seems mostly larger companies that are sticking with non-cloud hardware and services. The advantage of kubernetes in that case is that there is a large ecosystem of helm charts, guides, documentation, etc. Deploying something new from scratch is fairly easy since someone else has done all the leg work and packaged it all up.
  - rantwasp 6y ago
    just because something is packaged does not mean it’s usable. YMMV but this is how security horror stories start. Someone ran a container they had no idea where it came from, happily used a helm chart. Most of the times it’s not even malicious - it’s outdated software because “it just works”
    - marcinzm 6y ago
      In my experience all the common helm charts and docker images are regularly updated. If you don't update your installation of them then you also wouldn't update a docker compose or LXD.
      - rantwasp 6y ago
        that’s a big statement. what constitutes “common” helm charts?
- mplewis 6y ago
  For a different perspective, learning Kubernetes and using it widely gives you a universal set of tools for managing all sorts of applications at all scales. https://news.ycombinator.com/item?id=26502900 https://news.ycombinator.com/item?id=26502900
  - ForHackernews 6y ago
    Sure, this is kind of true, but only in the most depressing way possible. Kubernetes is overengineered and terrible but it's also just about the only game in town if you want a vendor-agnostic container orchestration system. This is the same situation as Javascript circa 2008: "Learning this absolute dogshit language and using it widely will give you the ability to write universal applications that will run in browsers everywhere and make you very employable." You're not wrong about k8s today, and wouldn't have been wrong about JS in the past, but boy is it a sad indictment of our industry that these things are true.
    - marcinzm 6y ago
      Kubernetes is engineered to solve the five hundred different problems that different people have in a way that works for them. If you don't do that then everyone will complain about their feature or some edge case being missing and not use the system (see comments on mesos in this thread). That's not over engineering, that's the required amount of engineering for this sort of system.
      - ForHackernews 6y ago
        Yeah, k8s people keep telling me this. But also, k8s "secrets" are not, in fact, secret, you can't actually accept traffic from the web without some extra magic cloud load balancer (cf https://v0-3-0--metallb.netlify.app/ https://v0-3-0--metallb.netlify.app/ maybe eventually) or properly run anything stateful like a database (maybe soon). Forget covering "everyone's" use cases: From where I'm sitting, k8s is an insanely complicated system that does a miserable job of covering the 95% use cases that Heroku solved in 2008. It's great that k8s (maybe) solves hard problems that nobody except Google has, but it doesn't solve the easy problems that most people have.
- 0xEFF 6y ago
  > For over 90% of workloads kubernetes is an overkill. It's not. Take any simple web app and deploy it into managed GKE with Anthos and you automatically get SLI's and the ability to define SLO's for availability and latency with a nice wizard. Takes a few minutes to setup. The amount of engineering needed to achieve good SLO monitoring dwarfs the engineering needed to run a simple app so it just never happened. That's no longer the case. > Only when company is reaching google scale kubernetes make sense. Also obviously not true given the number of companies deriving value from kubernetes. Note, I'm a Google Cloud partner.
  - dragonsh 6y ago
    Your statement already support that without the blessings of engineering team of Amazon, Google, Microsoft, Digital Ocean and various managed kubernetes service it's impossible for a reasonable small team to manage and monitor k8s and all of this service comes with lock-in and additional capital outlay. Obviously for a Google Cloud Partner, more people are tied to gcp and kubernetes, higher the revenue. Its secondary if it's really necessary for an application to require k8s.
    - deleted 6y ago
      [deleted]
- orthecreedence 6y ago
  I don't know about this. I caution people against microservices architecture all the time, but at my company having a scheduler just made sense. We spin up and down queue workers by massive amounts every hour, and doing this with anything besides a scheduler would be really tricky. Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.
  - p_l 6y ago
    It doesn't even have to be some microservices monster, just trying to bin-pack a bunch of different services, even monoliths, onto a fleet of servers. If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)
- lima 6y ago
  > For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. I hear people repeating this truism all day, but from practical experience, it doesn't seem to be the case - Kubernetes is paying dividends even in small single-node setups.
  - lovedswain 6y ago
    K8s single noder and GKE user here, can confirm I wouldn't remotely even consider going back. Deploying an app takes 5-10 minutes at most first time around, new pushes <1 minute, and when there is ops bullshit involved, it is never wasted work, and never needs to be repeated twice. I hated Kubernetes ops complexity at first, but there really isn't that much to it, and if it's too much, a service like GKE takes 70%+ of it away from you
- madhadron 6y ago
  > Only when company is reaching google scale kubernetes make sense. Long before Google scale. Kubernetes won't handle clusters anywhere near that large.
desktopninja 6y ago
Personally think AWS ECS is a 3rd place contender. Would add sprinkles to it if only they'd allow yaml files vs json configs in the aws-cli. ecs-cli and copilot are alright. Generally prefer to stay as close as possible to aws-cli
- kbar13 6y ago
  AWS CDK makes ECS a lot more palatable
- whatsmyusername 6y ago
  In AWS I’d choose ECS over any k8s variation on price alone before you get to the IAM integrations.
fastball 6y ago
Docker-swarm?
- mplewis 6y ago
  Docker Swarm is being cannibalized by k8s.
- yjftsjthsd-h 6y ago
  Having worked at a company running Docker swarm at... medium(?) scale... I have witnessed a truly shocking variety of bugs in its network stack. I always wondered if it was something specific to the company setup, but the result is that we just couldn't stay on a platform that would randomly fail to give containers a working network connection to each other.
ForHackernews 6y ago
Flockport? https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai...

zekrioca 6y ago

I'm sad for that.. It was truly a neat project with many great ideas, but Kubernetes influence brought it to an end :/

specialp 6y ago

We used Mesos as our first container orchestration stack. It worked OK but Kubernetes came along and offered a one stop solution for everything. In Mesos you had to use separate projects for container management (Marathon), discoverability (Consul/HAPROXY). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has. This was in 2014 so it was early but k8s came along with everything we needed.

kevincox 6y ago

I used mesos a while ago across a couple of cheep VPS for personal projects and services and it was great. I especially liked that I could used a shell executor to run services with filesystem access. For many environments this probably didn't make sense but for me I could put some secrets on the filesystem and services could make unix sockets available to nginx for SSL and hostname based forwarding. Also with Nix I could easily install software the specific services needed and it was trivial to integrate into the garbage collector making it much faster than launching a docker container. That being said the project wasn't moving that fast and it was a bit buggly for me and nodes would regularly get into loops where they couldn't authenticate with the master for various reasons (I think there were timing issues and timeouts that caused issues over the internet). Now I'm using Kubernetes but I have all of this complicated virtual networking that I don't need, I'm locked into docker-like containers and the thing is so complicated that I need to pay someone to run it for me.

xyzzy_plugh 6y ago

I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start. For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the competition. Mesos was trying to become a solution seemingly from scratch. I worked with and interacted with a number of Mesos project members and committers, and while they were generally bright folks, their industry experience was surprisingly shallow. In my experience, Mesos changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile. Within a year Kubernetes was waaaaay ahead and Mesos started considering pivoting to support k8s. I no longer see a purpose in Mesos, and frankly, that's okay. Too many Apache projects are on eternal life support, and they lower the bar for the rest of the Apache ecosystem, which has sort of begun to earn a poor reputation for quality (ala some of the fringe Boost libraries). Apache Foundation is no longer a label for solid, reliable software.

dharmab 6y ago
We used Mesos in production until 2020 (started transition to Kubernetes in 2018), and this comment is incredibly accurate. Mesos was an interesting project but the defaults were incredibly naive about production environments. Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'.
- bogomipz 6y ago
  >"Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'." What was Mesos default behavior when a node was powered off?
  - dharmab 6y ago
    It assumed that the Node had a temporary network fault and did not automatically remove the Node. Marathon also did not automatically schedule replacement containers on other Nodes. Kubernetes will, after a period of time, garbage collect the Node and schedule replacements as dictated by the controller.

johncena33 6y ago

How does this impact the company behind Mesos and products based on Mesos, e.g., Mesosphere etc.?

avita1 6y ago
Mesospehere was renamed and has shifted to supporting Kubernetes. https://d2iq.com/blog/mesosphere-is-now-d2iq https://d2iq.com/blog/mesosphere-is-now-d2iq
rad_gruchalski 6y ago
Mesosphere is now D2iQ, they have their own Kubernetes and stopped supporting Mesos / Marathon in October 2020.

kraemate 6y ago

Mesos made a ton of new contributions to distributed resource management. Resource offers and application integration can allow high cluster utilization and efficiency. Giving applications the opportunity to take part in resource allocation and scheduling was similar to the Exokernel design, and led to many interesting middleware architectures. Mesos also introduced Dominant Resource Fairness for allocating resources to competing applications, which has become an important concept in CS and specifically computational economics in its own right. Mesos achieved the rare combination of using CS research towards a widely deployed operational system. Its C++ code is simple enough to understand and hack, and seemed like one of those projects that you can/want to reimplement on your own "in a few days". It is sad how quickly it has become obsolete.

jeffbee 6y ago
Are there examples of high-utilization, large-scale Mesos deployments? Mesos didn't even gain over-commit until 2015, so it seems like it was generally behind the state of the art.
- dharmab 6y ago
  Most famously, Siri (used to?) run on a very large scale Mesos deployment (10000s of nodes, much higher than Kubernetes can scale to). Unfortunately the original article is lost, but here's a summary: https://daringfireball.net/linked/2015/04/29/siri-apache-mesos https://daringfireball.net/linked/2015/04/29/siri-apache-mes...
  - jeffbee 6y ago
    OK but what was the utilization? I'm not really sure K8s is state-of-the-art either. There are published research papers about very-large-scale clusters with 80%+ resource utilization.
    - dharmab 6y ago
      In our production experience, utilization had far more to do with the service owners (or autoscalers/auto-tuners) correctly choosing the cgroups and CPU scheduler allocations, as well as the kernel settings for cgroup slicing and CPU scheduler. We had Mesos clusters with 3% utilization and have Kubernetes clusters with 95%+ utilization. But we also have Kubernetes clusters with <10% utilization.

fnord123 6y ago

Fun fact: According to the initial Mesos paper: Spark was a demo project to show off Mesos. https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf > To validate our hypothesis that specialized frameworks providevalue over general ones, we have also built a new frame-work on top of Mesos called Spark ...

zekrioca 6y ago
Look at the authors list. Some of them also participated in creating and designing Spark.
- fnord123 6y ago
  Yes, this is not a surprise. The fun fact part is that Spark began as merely a way to validate the hypothesis of Mesos and look how the hypothesis validation project has taken off. Neat!

yuppie_scum 6y ago

RIP. It was a strong contender for a while before Kube matured.

jeffbee 6y ago

I wonder what recent large-scale adopters of Mesos are going to do. I know Dropbox deployed Mesos in the last year or so[1], even though it was pretty obvious that the project community was dead. Will the existing users found a new community? https://www.amd.com/system/files/documents/new-epyc-case-study-dropbox.pdf https://www.amd.com/system/files/documents/new-epyc-case-stu...

benjaminwootton 6y ago

I went to see Mesos early in their life in their San Francisco office after a joint customer put us in touch. Never in my life did I meet such an arrogant group of people. First, they left us waiting in reception for an hour. They eventually took the meeting over lunch, where we had to watch them inhaling their free food. Some guy in plastic-leather trousers spent most of the hour lecturing us about all of the multi million deals they were doing, and how they weren't interested in speaking with anyone who couldn't write 7 figure checks. Not once did they ask about our business or even our names if I remember correctly. I had a similar experience with other west coast tech companies. Too arrogant for their own good and not able to put in the hard yards with stodgy old enterprise companies even when they have good technology and are early to market.

eklitzke 6y ago
You're thinking of Mesosphere, a separate company building a product on top of Mesos.
- benjaminwootton 6y ago
  True. As the major corporate sponsor though they didn’t give it the best chance. I ran one of the first Docker partners and saw first hand how both Mesos and Docker Inc had considerable mindset and interest from large enterprises for a year or so before Kubernetes matured. They both spectacularly failed to deliver through arrogance and bad execution.
  - pm90 6y ago
    I remember visiting SFBA in 2014 or so, a friend worked in PayPal said they used mesos, kubernetes was a baby project with a lot of hype. It sure seemed mesos had won back then, and the fact that kubernetes caught up, surpassed and then succeeded so wildly would have never crossed my mind.
    - smarterclayton 6y ago
      I worked with tech companies and some banks in the early days of Kubernetes who were deep in Mesos and many of them felt the same way. It’s really hard to see this as it’s happening. Kubernetes is absolutely “worse is better” when it comes to scheduling. But scheduling isn’t the problem most people needed solved. Kubernetes standardized deployment patterns of multiple apps well for most people. Everything else was a nice to have (scheduling, scale, ease of initial setup, etc).

ksec 6y ago

I remember at MesosCon, Apple said they are the largest Mesos user. I wonder if they also moved to K8s as well.

thor24 6y ago
They have apparently hired a ton of great k8 folks who have been tasked with building an internal layer of sorts for themselves so I guess they are moving off it (though not anytime soon).
Conan_Kudo 6y ago
Apple was at KubeCon last year, so I assume they've moved to Kubernetes too.

fh973 6y ago

The framework-based scheduler architecture looked like an interesting concept at first sight. But its advantage (write your custom scheduling policy, along with gaining scalability) evidently wasn't that important to make up for the effort of actually having to write your own framework to run anything. (In this regard its a bit like object storage as a file system replacement narrative, which promises scalability, if you adapt your application). This seems to be true both for the Omega experiments at Google (which I assume still uses Borg), and for Mesos. In the end, Mesos always lacked even a minimal application deployment story. Marathon unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API. I think if Mesos(-sphere) had recognized that gap in time, and came up with a decent framework for application deployments, instead of telling everyone to write their own framework (in C++, against a hacky code base), Kubernetes might not have had a chance, or at least we would have two alternatives to chose from. Too bad.

jaaron 6y ago

I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning: k8s is the Rails of the cloud. Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterprise web frameworks, Rails felt like a toy. It didn't have the features, robustness, safety, and scalability of "proper" frameworks. What did Rails do? It was easy to get started and it hid a lot of the boring and painful work of web frameworks at the time. Through the sheer force of will of a massive community, Rails grew up and became something more the toy it started as. I was pretty arrogant in my opinions about the Rails community at first, but then I ended up working on several Rails projects over the years. It still hides a lot under the hood, there are still arguably better technical frameworks out there and plenty of folks use it improperly, when they don't need to, and without really understanding the fundamentals, meaning that they tend to get in trouble when pushing the limits or moving outside the golden path of development. And I feel the same way about k8s. I think it started out without anywhere near the features of similar frameworks. It didn't scale well, was simplistic in it's model, and overly complicated in it's implementation. But it was much more approachable than something like Mesos and answered the question of "why am I containerizing everything?", giving everything a purpose to those who started down the path of Docker. And now it has a huge following and industry behind it. At this point, I've learned that what becomes popular isn't necessarily the "right" or "correct" architecture. There's a lot more to programming trends (and fads) than that. The whole industry seems to want to reinvent the wheel every decade, almost like some sort of planned obsolescence to justify our work. Nevertheless, it's rarely wise to fight against the tide and when you have the enough of the industry moving in a direction, we can make even toys into real tools.

meatmanek 6y ago
I actually think Kubernetes is a better foundation for a 2-level scheduler system than Mesos is. (In k8s land, they call this the operator pattern[1]). Each operator creates Pod objects in k8s with constraints/affinity/anti-affinity, and the K8s scheduler decides on your operator's behalf where each pod will go. Pending pods (pods that aren't assigned to boxes yet) are also a really useful signal for cluster autoscaling that is annoying to calculate in Mesos. (The Mesos master has no idea how many pods each framework _wants_ to launch, so we ended up writing code that knows how to deduce the resource demand from each framework's API or UI.) On the other side of the same coin, the framework in Mesos has no idea the total resources available to the cluster - it has to wait to be offered each box's resources in turn. This usually means frameworks either: a) accept the first offer which matches the basic requirements for a task, even if there are better places to run that task, or b) accept every single offer, under the assumption that they're the only framework on the cluster, and then implement their own scheduler on top of this pool of resources. (I call this the "monoframework" approach.) The former approach is how Mesos is supposed to work, but can lead to all sorts of bad outcomes, e.g. all the copies of a Marathon service running on the same box, or Spark having to use timeouts to know when it should give up waiting for offers if it doesn't get enough right away. The latter approach can lead to better placements, but defeats the purpose of the two-level scheduler, as no other frameworks can use the resources that aren't being used by the monoframework. Under Kubernetes, any operator can query the state of the cluster and make informed decisions about what to request. 1. https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...

throwaway823882 6y ago

Most of the comments here about Mesos are exactly my experience with Kubernetes. Replacing the words and I'm just nodding my head: "Kubernetes changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile." "Kubernetes was an interesting project but the defaults were incredibly naive about production environments." "Kubernetes unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API." "A challenge with Kubernetes is that Kubernetes was a piece of technology, a framework at most, instead of a product." "In Kubernetes you had to use separate projects for container management (kubelet, kube-scheduler, kube-controller-manager, kube-proxy, cri-o), discoverability (etcd). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has."

dharmab 6y ago
As someone you quoted: Take every problem you have with Kubernetes, multiply by 10x to 100x, and you have DC/OS. It's all perspective :)

random3 6y ago

It's sad, but expected. This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc. Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to. Kubernetes was supposed to be scheduler for Mesos and Google invested in it. Let's not forget John Wilkes (Google Borg, Omega, Kubernetes) on the stage at MesosCon in 2014 https://www.youtube.com/watch?v=VQAAkO5B5Hg https://www.youtube.com/watch?v=VQAAkO5B5Hg While I don't know the exact reasons behind Kubernetes becoming the scheduler and resource manager, I think that has very much to do with the stewardship of the Mesos project as well. By 2015/16 most Mesos committers were at Mesosphere. Suffice to say the open-governance was more of pain than a benefit to them. If you were at Apple, Adobe and others that relied on Mesos you didn't have much to say. Everything shifted towards DCOS and everyone was pushed towards commercial licenses. This is said, because sadly Kubernetes didn't fix the whole problem and left us with a distributed system with a text based "API" that requires text patching to manage and borderline clueless about the lifecycle of the services running on top. Yet, it's the best we got :)

jaaron 6y ago
This is much more the real story but I doubt most folks will ever hear it. Totally agree. Mesos project always had a trouble with governance and didn't build out the larger community the way other projects did. If they had built that coalition, made the project more accessible to others, who knows, maybe it would have gone differently. Then again, k8s succeeded in part due to the Google reputation (even if undeserved) and it's use of GoLang. I always found Mesos' use of C++ meant many of the users (often developing in Java, Go, or Node) just wouldn't contribute.
- aabhay 6y ago
  I think the problem went deeper than that actually. Mesos required that applications use a distributed system API to interface with the executors. This meant it was very challenging to write a distributed system upon Mesos, even though that was ostensibly the very thing that Mesos was meant to solve. When K8s came along and made everything manifest and YAML-based, we sort of knew that we were screwed because of how much simpler that was.

relistan 6y ago

The real gem of the Mesos ecosystem was the much lesser known Singularity scheduler from HubSpot. https://github.com/HubSpot/Singularity https://github.com/HubSpot/Singularity I've run this at scale, in production since 2015 and it has been absolutely rock solid and does most of the production things you'd want. Unlike commercial products, it was written to run HubSpot's own infrastructure so it does what a production system needs. Really bummed to have to downgrade to something like K8s in the near future.

tpetr 6y ago
Thanks for the kind words! Singularity is still a core piece of infrastructure for running stateless apps at HubSpot (our status page currently reads ~13k Requests, ~22k Active Tasks, ~700 Agents). We're also heavily investing in Kubernetes, but Singularity is solid enough that our focus has been more towards reliably running and scaling stateful things like MySQL, ZK, Kafka, HBase, and ES.
chucky_z 6y ago
There’s a Singularity driver for Nomad https://www.nomadproject.io/docs/drivers/external/singularity https://www.nomadproject.io/docs/drivers/external/singularit... Nomad is pretty darn easy to run and scales far. I help run about 20 clusters in day job from 100 to 1800 nodes, we’re going to push one of them to over 8500 nodes here soon too. We don’t use the Singularity driver but we wrote two of our own and they work well and are easy to maintain.

aggress 6y ago

Disclaimer - I used to work for Mesosphere. On the topic of arrogance, I wasn't in that meeting referenced, but having worked with most people there who could have been, and being a Brit who's worked for west coast startups like that for nearly 10 years, it's that American confidence, outlook and drive (along with Sand Hill road) that contributes to so much success out there. I'm a fan, but have to check myself every time i use 'super' as an adjective. Tech vs Product is a good comparison to make. I banged my head many times on DC/OS (the commercial version of Mesos) and underlying Mesos itself. Solid tech, but I'd have liked to have seen the product develop faster, to realise its potential in the enterprise market, which was huge.

benjamin_mahler 6y ago

I’m one of the long term PMC / committers on mesos. In retrospect I feel this was inevitable to a few key reasons: * k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster management systems. We learned many things along the way that we would do differently a second time around. * Mesos was too “batteries not included”: this fragmented the community, made it unapproachable to new users, and led to a lot of redundant effort. Most users just want to run services, jobs and cron jobs, but this was not part of Mesos and you had to choose from the array of ecosystem schedulers (e.g. Aurora, Marathon, Chronos, Singularity, etc) or building something yourself. * Mesosphere was a VC backed startup and drove the project after Twitter. Being a VC backed startup you need to have a business model that can generate revenue. This led to a lot of tensions and mistrust with users and other vendors. Compare this with Google / k8s, where Google does not need to generate revenue from k8s directly, it can instead invest massive amounts of money and strong engineers on the notion that it will improve Google’s cloud computing business. Even if k8s hadn’t come along, Mesos was ripe to be disrupted by something that was easier to use out of the box, and that had a benevolent home that could unite vendors and other contributors. Mesos could have perhaps evolved in this direction over time but we'll never know now.

rad_gruchalski 6y ago
Thanks for all the work!
qbasic_forever 6y ago
It's refreshing to see an honest and critical evaluation of things. This kind of decision should be celebrated and encouraged with all projects.
- waynesonfire 6y ago
  On the announcement that it's getting shelved you get a retro. How is that refreshing? Whats your baseline? Where is D2iQ response to this news?

ta20210405 6y ago

Somewhat of an end of an era. Worked at a shop that had one of the largest Mesos fleets in the world. Thousands of physical nodes, petabytes of data. True Mesos in production was something to witness.

lars_francke 6y ago

Nothing is decided yet. This is a vote thread which _might_ end with someone interested in Mesos stepping in to pick the project up. This hasn't happened in a while with other projects (a whole bunch of Apache projects have just retired in the last few months) but with Mesos there might be a chance.

throwaway_1237 6y ago

What is the attic?

arnon 6y ago

The vote has been cancelled. New discussion - https://news.ycombinator.com/item?id=26748191 https://news.ycombinator.com/item?id=26748191