3 ms·

I think it's a matter of scope. For some relatively small projects, k8s is sufficient. But pretty quickly you end up needing more and more functionality and yo

by jaaron 10y ago

I think it's a matter of scope. For some relatively small projects, k8s is sufficient. But pretty quickly you end up needing more and more functionality and you end up in a larger stack like OpenShift. That's the place that Mesos and DC/OS (IMHO) shine: when you're working with large clusters (1000+) running very different workloads. Because in that scenario, k8s is still pretty immature and you're going to inevitably need to solve all the other problems that OpenShift, DC/OS, Rancher or other stacks solve. Just like Linux distros, in theory, anyone can do it. In practice, it's a pain and you want to pick up a standard distro. That's what these stacks provide: a pre-configured suite of open source tools that get you way more than k8s or Mesos or Marathon on their own provide.

lowbloodsugar 10y ago
Sure, but DC/OS has yet to demonstrate that it can do any of that. Mesos has demonstrated that it can handle clusters at scale, but I don't know of anyone running the full smorgasbord of all 30 services on a cluster that big, and its the other 29 things that's going to fail at scale. And again, if you are going to be running DC/OS at that scale, then you are going to be running the enterprise version, because you aren't going to be running that much magical shit on million dollar hardware without someone being paid to troubleshoot. That might be me in a few years, except that Mesosphere seems to be attempting to kill me in the short term by killing non-DC/OS mesos/marathon while crippling non-enterprise DC/OS. So looks like we'll be migrating to K8s and hoping that in a few years K8s scales. The "K8s doesn't scale" argument is losing ground with every new release btw.
- hartem_ 10y ago
  What makes you feel that non-enterprise DC/OS is crippled in any way? Disclaimer: I work at Mesosphere.
  - lowbloodsugar 10y ago
    Auth.
- jadbox 10y ago
  All of Siri and Azure runs on top of DC/OS, it's certainly demonstrated it can be used for massive data centers
  - balakk 10y ago
    All of Azure ? :) DC/OS is a moderately popular workload on Azure, yes. Disclaimer: I work for MS
  - ninkendo 10y ago
    Siri doesn't run on DC/OS. They use mesos (ie. the open source apache foundation project that marathon and later DC/OS are built on top of), but write their own framework(s) that run on top of it. Source: I worked on the team, although we had talked about this at mesoscon in the past.
- jgehrcke 10y ago
  > but DC/OS has yet to demonstrate that it can do any of that [...] its the other 29 things that's going to fail at scale. Of course you are right that in DC/OS there is a lot more besides Mesos that could fail at scale. That is why we are carefully scale-testing DC/OS in its entirety -- under lab conditions (internally) as well as under real-world conditions (large production environments). We will be more transparent about this in the future, but one example I can give is that we regularly run DC/OS on 10^3 nodes for testing purposes. > if you are going to be running DC/OS at that scale, then you are going to be running the enterprise version I am not convinced by that argument. Still, just to clarify, Mesosphere Enterprise DC/OS does not have better scaling characteristics than (open) DC/OS. We are trying to land all corresponding goodness in DC/OS. > killing non-DC/OS mesos/marathon I understand that brutal decisions have been made. But if you look at Mesos and Marathon as of today (btw., there is no such thing as "non-DC/OS mesos/marathon"), progress is being made at an impressive rate. I observe more "animating" than "killing". > while crippling non-enterprise DC/OS I only see additions, no removals. Also, if you were looking at the core technology stack, the delta between Enterprise DC/OS and DC/OS will probably appear to be surprisingly small to you. > The "K8s doesn't scale" argument is losing ground with every new release btw. I agree. The "scale argument" is insignificant compared to other aspects in most of the cases anyway. As we all know :-).

by jaaron 10y ago

lowbloodsugar 10y ago

Sure, but DC/OS has yet to demonstrate that it can do any of that. Mesos has demonstrated that it can handle clusters at scale, but I don't know of anyone running the full smorgasbord of all 30 services on a cluster that big, and its the other 29 things that's going to fail at scale. And again, if you are going to be running DC/OS at that scale, then you are going to be running the enterprise version, because you aren't going to be running that much magical shit on million dollar hardware without someone being paid to troubleshoot. That might be me in a few years, except that Mesosphere seems to be attempting to kill me in the short term by killing non-DC/OS mesos/marathon while crippling non-enterprise DC/OS. So looks like we'll be migrating to K8s and hoping that in a few years K8s scales. The "K8s doesn't scale" argument is losing ground with every new release btw.

hartem_ 10y ago
What makes you feel that non-enterprise DC/OS is crippled in any way? Disclaimer: I work at Mesosphere.
- lowbloodsugar 10y ago
  Auth.
jadbox 10y ago
All of Siri and Azure runs on top of DC/OS, it's certainly demonstrated it can be used for massive data centers
- balakk 10y ago
  All of Azure ? :) DC/OS is a moderately popular workload on Azure, yes. Disclaimer: I work for MS
- ninkendo 10y ago
  Siri doesn't run on DC/OS. They use mesos (ie. the open source apache foundation project that marathon and later DC/OS are built on top of), but write their own framework(s) that run on top of it. Source: I worked on the team, although we had talked about this at mesoscon in the past.
jgehrcke 10y ago
> but DC/OS has yet to demonstrate that it can do any of that [...] its the other 29 things that's going to fail at scale. Of course you are right that in DC/OS there is a lot more besides Mesos that could fail at scale. That is why we are carefully scale-testing DC/OS in its entirety -- under lab conditions (internally) as well as under real-world conditions (large production environments). We will be more transparent about this in the future, but one example I can give is that we regularly run DC/OS on 10^3 nodes for testing purposes. > if you are going to be running DC/OS at that scale, then you are going to be running the enterprise version I am not convinced by that argument. Still, just to clarify, Mesosphere Enterprise DC/OS does not have better scaling characteristics than (open) DC/OS. We are trying to land all corresponding goodness in DC/OS. > killing non-DC/OS mesos/marathon I understand that brutal decisions have been made. But if you look at Mesos and Marathon as of today (btw., there is no such thing as "non-DC/OS mesos/marathon"), progress is being made at an impressive rate. I observe more "animating" than "killing". > while crippling non-enterprise DC/OS I only see additions, no removals. Also, if you were looking at the core technology stack, the delta between Enterprise DC/OS and DC/OS will probably appear to be surprisingly small to you. > The "K8s doesn't scale" argument is losing ground with every new release btw. I agree. The "scale argument" is insignificant compared to other aspects in most of the cases anyway. As we all know :-).