5 ms·

If you've spent any significant amount of time with Mesos and Marathon, you've probably found that it's buggy (Marathon at least), complicated, and hard to work

If you've spent any significant amount of time with Mesos and Marathon, you've probably found that it's buggy (Marathon at least), complicated, and hard to work with. Zookeeper is just one part of the complicated web of infrastructure you need to stand up. To top it off you end up needing to run something like Netflix' Exhibitor to have any shred of confidence that if ZK goes away the state of your cluster will be recoverable. It definitely "works", but certainly leaves something to be desired in the stability and ease of use department.

otterley 12y ago
I'm curious about just how difficult managing ZK is. My initial experience was pretty pleasant. Sure, it doesn't have a pretty admin UI (Exhibitor fills in that gap), but it's not like the consensus is that ZK is unreliable or doesn't live up to its utility claims. So, what exactly am I missing here?
- phildougherty 12y ago
  I wasn't picking on ZK in particular. I was more pointing out that the system as a whole is complicated. It might be fine for "enterprise deployments" where a company has a lot of time, money and resources to throw at the implementation. In my opinion the average end user is going to get lost in the details of how these systems interconnect and work together. Debugging some of the more complex issues that they will face will be challenging. Even Exhibitor is not a silver bullet for managing ZK. It helps with rolling restarts and lets you do a sort of MySQL-esque bin-log replay of your state, but even with Exhibitor, it's not going to make for a smooth recovery at 4:00AM when PagerDuty rings. The underlying design of how Mesos and Marathon communicate create situations where they have differing views of the state of the cluster, and you end up with "orphans" which are tasks that Mesos is aware of but Marathon knows nothing about. In my opinion, the system feels like it was designed to be something else, and then all this functionality was tacked on later. Coincidentally that is exactly what the case is with Mesos. I think there is a lot of room for an improved user experience that these tools will struggle to provide.
  - presspot 12y ago
    The whole point of the Mesosphere DCOS product is to make Mesos consumable by mere mortals: https://www.youtube.com/watch?v=UgJMlHdZEx4 https://www.youtube.com/watch?v=UgJMlHdZEx4
nor0101 12y ago
Hi there, Marathon committer here. If you had a bad experience with the 0.7.x series, I apologize for that. We've made a serious effort to fix the consistency issues and that work has paid off. 0.8.0 is current, and we've just tagged 0.8.1-RC1 yesterday. I encourage you to give it another try! As mentioned in the other comments, we (Mesosphere) are actively working to provide alternatives to zookeeper such as etcd. The reality is that ZK will stick around as a stable option for people, as the first > 1.0 etcd release was tagged only two months ago.
23david 12y ago
Yep. In my experience, I've seen that the issue isn't with Mesos the Apache project, but rather with Marathon and Chronos, which are the applications built by Mesosphere. There's a big naming and marketing confusion at the moment with the open-source project and the VC-funded startup having such a similar name. It's unfortunately giving a bad reputation to the Mesos project. Curious why there actually isn't a trademark issue here...?
- phildougherty 12y ago
  Because Mesosphere (the VC backed startup) hired the creator of Mesos (the Apache project), and is now trying to capitalize on the open source project as one of their "products".
  - presspot 12y ago
    Mesosphere is arguably the largest contributor to Mesos, certainly on par with Twitter, especially when you consider all the surrounding ecosystem. The company also secured permission from the Apache foundation with the trademark when the company was founded. It's good for the open source ecosystem to have companies productize and support projects, particularly when they are plowing millions of dollars back into the open source.
- eropple 12y ago
  FWIW, I've had much more success with HubSpot's Singularity than Marathon/Chronos. (I use Marathon to bootstrap Singularity as the "init system", because it had HA before Singularity, but that's the extent of its interaction with my system.)
  - phildougherty 12y ago
    That's interesting. I've also found it interesting that one of the big selling points of the Mesosphere ecosystem is that it is used at scale by Twitter. But AFAIK twitter developed their own framework for long running tasks (http://aurora.incubator.apache.org/ http://aurora.incubator.apache.org/). So they are using Mesos (the open source project), maybe Chronos, and their own version of Marathon. That can be somewhat misleading to newcomers.
maslam 12y ago
We[1] haven't found this to be the case. These are early-stage projects but they hold promise for companies that need an internal PaaS at scale. I look at Marathon, Chronos and ZK (to some degree) as building blocks, and DCOS as the glue that will tie them together. One big reason to use Marathon + Chronos + Mesos is to get multi-tenancy of compute and data workloads on your instances. You can't get that with web-service specific tech like Tutum. [1] appuri.com
- josh2600 12y ago
  I mean, I'm a little biased because I work there, but Terminal gives you multi-tenancy of compute and data workloads by default, and you get almost all of the properties of VMWare on containers (including VMotion style migrations). You can make an account and boot apache Spark in about 30 seconds using this link [0]. It's running in production right now for a lot of people, and you can run Mesos on top of Terminal if you want it [1]. Again, I'm not trying to push this on you if you're happy with how stuff works today, but I think we've made a PaaS that solves a lot of these problems (and we'll let you run it on your own metal too if you want it). Check it out at terminal.com if you have some free time. [0]https://www.terminal.com/snapshot/c81e6215eba5799335a45b69360d14e26bac50358c369066d15b159ac704a33d https://www.terminal.com/snapshot/c81e6215eba5799335a45b6936... [1]https://www.terminal.com/snapshot/44d4ee043422afec75dfd3bdaa6428ad3ea3aea72607a93aadc3483b7d5d9567 https://www.terminal.com/snapshot/44d4ee043422afec75dfd3bdaa...
  - mdekkers 12y ago
    This post would be a lot less spammy if you actually added some content, like how you do things differently, or what stack you use, whatever. Anything except "look at us, we are great at this"
    - josh2600 12y ago
      We hacked the linux kernel to provide better checkpointing of RAM state and then we write the state to disk. Anyone can then summon the state from disk in the time it takes to read the SSD. It's kinda cool, or at least I think so.
hawk188 12y ago
Zookeeper isn't complicated. Maybe you picked the wrong profession?
- sitkack 12y ago
  When ur green and have a karma of one, maybe u shouldn't try and append some junk on the insult log, hawk.
florianleibert 12y ago
I'd love to hear about the bugs that you encountered. Did you file any github issues for them?

otterley 12y ago

I'm curious about just how difficult managing ZK is. My initial experience was pretty pleasant. Sure, it doesn't have a pretty admin UI (Exhibitor fills in that gap), but it's not like the consensus is that ZK is unreliable or doesn't live up to its utility claims. So, what exactly am I missing here?

phildougherty 12y ago
I wasn't picking on ZK in particular. I was more pointing out that the system as a whole is complicated. It might be fine for "enterprise deployments" where a company has a lot of time, money and resources to throw at the implementation. In my opinion the average end user is going to get lost in the details of how these systems interconnect and work together. Debugging some of the more complex issues that they will face will be challenging. Even Exhibitor is not a silver bullet for managing ZK. It helps with rolling restarts and lets you do a sort of MySQL-esque bin-log replay of your state, but even with Exhibitor, it's not going to make for a smooth recovery at 4:00AM when PagerDuty rings. The underlying design of how Mesos and Marathon communicate create situations where they have differing views of the state of the cluster, and you end up with "orphans" which are tasks that Mesos is aware of but Marathon knows nothing about. In my opinion, the system feels like it was designed to be something else, and then all this functionality was tacked on later. Coincidentally that is exactly what the case is with Mesos. I think there is a lot of room for an improved user experience that these tools will struggle to provide.
- presspot 12y ago
  The whole point of the Mesosphere DCOS product is to make Mesos consumable by mere mortals: https://www.youtube.com/watch?v=UgJMlHdZEx4 https://www.youtube.com/watch?v=UgJMlHdZEx4

nor0101 12y ago

Hi there, Marathon committer here. If you had a bad experience with the 0.7.x series, I apologize for that. We've made a serious effort to fix the consistency issues and that work has paid off. 0.8.0 is current, and we've just tagged 0.8.1-RC1 yesterday. I encourage you to give it another try! As mentioned in the other comments, we (Mesosphere) are actively working to provide alternatives to zookeeper such as etcd. The reality is that ZK will stick around as a stable option for people, as the first > 1.0 etcd release was tagged only two months ago.

23david 12y ago

Yep. In my experience, I've seen that the issue isn't with Mesos the Apache project, but rather with Marathon and Chronos, which are the applications built by Mesosphere. There's a big naming and marketing confusion at the moment with the open-source project and the VC-funded startup having such a similar name. It's unfortunately giving a bad reputation to the Mesos project. Curious why there actually isn't a trademark issue here...?

phildougherty 12y ago
Because Mesosphere (the VC backed startup) hired the creator of Mesos (the Apache project), and is now trying to capitalize on the open source project as one of their "products".
- presspot 12y ago
  Mesosphere is arguably the largest contributor to Mesos, certainly on par with Twitter, especially when you consider all the surrounding ecosystem. The company also secured permission from the Apache foundation with the trademark when the company was founded. It's good for the open source ecosystem to have companies productize and support projects, particularly when they are plowing millions of dollars back into the open source.
eropple 12y ago
FWIW, I've had much more success with HubSpot's Singularity than Marathon/Chronos. (I use Marathon to bootstrap Singularity as the "init system", because it had HA before Singularity, but that's the extent of its interaction with my system.)
- phildougherty 12y ago
  That's interesting. I've also found it interesting that one of the big selling points of the Mesosphere ecosystem is that it is used at scale by Twitter. But AFAIK twitter developed their own framework for long running tasks (http://aurora.incubator.apache.org/ http://aurora.incubator.apache.org/). So they are using Mesos (the open source project), maybe Chronos, and their own version of Marathon. That can be somewhat misleading to newcomers.

maslam 12y ago

We[1] haven't found this to be the case. These are early-stage projects but they hold promise for companies that need an internal PaaS at scale. I look at Marathon, Chronos and ZK (to some degree) as building blocks, and DCOS as the glue that will tie them together. One big reason to use Marathon + Chronos + Mesos is to get multi-tenancy of compute and data workloads on your instances. You can't get that with web-service specific tech like Tutum. [1] appuri.com

josh2600 12y ago
I mean, I'm a little biased because I work there, but Terminal gives you multi-tenancy of compute and data workloads by default, and you get almost all of the properties of VMWare on containers (including VMotion style migrations). You can make an account and boot apache Spark in about 30 seconds using this link [0]. It's running in production right now for a lot of people, and you can run Mesos on top of Terminal if you want it [1]. Again, I'm not trying to push this on you if you're happy with how stuff works today, but I think we've made a PaaS that solves a lot of these problems (and we'll let you run it on your own metal too if you want it). Check it out at terminal.com if you have some free time. [0]https://www.terminal.com/snapshot/c81e6215eba5799335a45b69360d14e26bac50358c369066d15b159ac704a33d https://www.terminal.com/snapshot/c81e6215eba5799335a45b6936... [1]https://www.terminal.com/snapshot/44d4ee043422afec75dfd3bdaa6428ad3ea3aea72607a93aadc3483b7d5d9567 https://www.terminal.com/snapshot/44d4ee043422afec75dfd3bdaa...
- mdekkers 12y ago
  This post would be a lot less spammy if you actually added some content, like how you do things differently, or what stack you use, whatever. Anything except "look at us, we are great at this"
  - josh2600 12y ago
    We hacked the linux kernel to provide better checkpointing of RAM state and then we write the state to disk. Anyone can then summon the state from disk in the time it takes to read the SSD. It's kinda cool, or at least I think so.

hawk188 12y ago

Zookeeper isn't complicated. Maybe you picked the wrong profession?

sitkack 12y ago
When ur green and have a karma of one, maybe u shouldn't try and append some junk on the insult log, hawk.

florianleibert 12y ago

I'd love to hear about the bugs that you encountered. Did you file any github issues for them?