10 ms·

Introduction to Apache Mesos

145 points by aant 12y ago

preillyme 12y ago
I'm curious which of the Marathon bugs caused your downtime?
- aant 12y ago
  I'm afraid I don't remember exactly which one it was but after an upgrade of the framework it worked perfectly again.
  - florianleibert 12y ago
    The 0.7X releases contained a number of bugs. 0.8X has been significantly better!
- phildougherty 12y ago
  Don't poll the apps API if you have a decent number of tasks or it can bring the whole system to its knees.
wmf 12y ago
Maybe this is a good place to ask a question that I've been pondering for a while: Why is Mesos based on the concept of resource offers? AFAIK this is backwards compared to other schedulers where you ask for resources and the schedule gives them to you (or not).
- aant 12y ago
  I can't quite tell you why it was implemented like this. The Omega paper by Google describes different kinds of scheduling algorithms and their pros and cons. I don't think that it's backwards compared to other schedulers (many of them are monolithic) but I think it can be improved by using transactions and optimistic concurrency control. This is discussed in the Omega paper and I think there's work to integrate the shared state scheduler into Mesos.
- sabraham 12y ago
  To load the framework (application) with the logic with the concern of deciding what resources it wants. The scheduler shouldn't care about what you get, just fairness. Two-tiered scheduling achieves this: Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. http://mesos.berkeley.edu/mesos_tech_report.pdf http://mesos.berkeley.edu/mesos_tech_report.pdf
  - wmf 12y ago
    I guess that makes sense if frameworks care about placement (which I generally don't) because otherwise they'd have to express placement constraints to Mesos. I still can't find documentation about what fairness policies are actually implemented, though.
    - sabraham 12y ago
      Mesos uses dominant resource fairness: https://www.cs.berkeley.edu/~alig/papers/drf.pdf https://www.cs.berkeley.edu/~alig/papers/drf.pdf
- otterley 12y ago
  The most puzzling thing for me is why schedulers and executors are so tightly coupled in Mesos. In my mind, those are separate concerns: schedulers should concern themselves solely with locating an available resource for you (and should be part of the Mesos core), and the executor should take care of launching and managing the program to be run. With the current design, if I want to launch a Docker container, I have to use its scheduler, even though its scheduler might not be as good as some other resource's scheduler. And I might have to accept a regression in the scheduler in order to benefit from a bugfix in the executor.
domenp 12y ago
Allow me take the opportunity and ask if anybody here is running Elasticsearch with Mesos (and Marathon) using Docker container? I'm running Elasticsearch nodes on a dedicated Mesos slaves and I’m still not sure how much memory should I allocate to Elasticsearch task e.g. all available memory as reported by Mesos or some smaller amount to leave something for the system? Please note that I'm asking about memory allocated to the Marathon task not about JVM heap size.
- nemothekid 12y ago
  I've taken the view of treating our database slaves as special and allocating all the memory on the slave to ES. That way ES (or any other db) always runs on the right slave, and no other potentially long running/resource hungry process runs alongside it (but just enough that's jenkins tasks and chronos tasks can still rune if need be). AKAIK, there isn't a right way to run databases because the persistent storage layer into mesos is still being baked.
  - dingdingdang 12y ago
    > the persistent storage layer into mesos is still being baked That's pretty important point, would have thought persistent storage would have been first point to get working for a project like Mesos. Also, their homepage ( http://mesos.apache.org/ http://mesos.apache.org/ ) outright states: "Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines [...]". What storage are they talking about if not persistent storage?
    - nemothekid 12y ago
      This mesos 0.22 talk goes into a bit more what they are imagining - http://mesosphere.com/2015/03/27/mesos-0-22-0-released/ http://mesosphere.com/2015/03/27/mesos-0-22-0-released/ Essentially the current mesos disk quotas are just so that tasks that run, run with a minimum amount of disk space. I think what they are trying to accomplish in future releases is some way you could build an EBS Style disk space management on top of mesos.
twoy 12y ago
If you don't want to manage ZooKeeper manually to bootstrap Mesos cluster, you need another Mesos cluster haha.
- saryant 12y ago
  You could run it on top of something like CoreOS, but then you're just swapping Zookeeper for etcd.
- eropple 12y ago
  Or just use Exhibitor. I don't remember the last time I manually anything'd with Zookeeper. Between Exhibitor and Curator, I honestly find Zookeeper so straightforward and easy to work with that I don't quite understand the popularity of etcd.
  - toomuchtodo 12y ago
    I'm running a small Exhibitor/Zookeeper cluster dockerized (5 nodes, several hundred clients), and its extremely straightforward. Etcd isn't what we'd consider production-grade yet.
    - saryant 12y ago
      Having worked with etcd in production for the last few months, I have to agree. The CoreOS stack needs some more time to marinate.
      - toomuchtodo 12y ago
        Thanks for this comment. Glad to know I made the right choice. I'm not saying etcd won't ever overshadow Zookeeper, it probably will with the momentum behind it, but as an ops guys, I wasn't willing to bet production application service discovery on it.
        eropple 12y ago
        My distaste for the Go community is pretty well-established in these parts; I think worse-is-better is screwing us all, and etcd seems to me to be the worse-is-better Zookeeper. And for things that don't matter, sure, worse-is-better your life away; a Rails app can be whatever you want, but the infrastructure I manage had better be bulletproof. I won't say etcd will never be competitive, but without some significant changes, I don't see it getting my vote--and those changes are largely around the parts of the feature set that etcd doesn't support, at which point...why use it, anyway?
rckrd 12y ago
An interesting talk about Docker and Mesos by a former coworker, he also contributed to 0.22: https://www.youtube.com/watch?v=ZZXtXLvTXAE https://www.youtube.com/watch?v=ZZXtXLvTXAE
andyidsinga 12y ago
question / thought : seems like mesos could evolve to be an alternative to openstack ...especially if a tenant layer is developed ..yes/no ???
- wmf 12y ago
  Probably. And OpenShift 3.0 (also on the front page today) is a PaaS (Cloud Foundry competitor) built on Mesos.
  - josephjacks 12y ago
    OpenShift is most definitely not built on Mesos. RedHat's plans over time are to integrate with Mesos at a lower level for fine grained resource allocation, but for now Kubernetes is used as the cluster scheduling and orchestrations runtime.
StavrosK 12y ago
Okay, maybe I'm getting old, but I can't really see the point of Mesos. It feels like it's adding too much magic to the mix, and I can't see myself using it to deploy web app servers/databases. Is there a different intended purpose that I'm missing? I suspect it's great for deploying task workers, for example.
- bonobo3000 12y ago
  Yeah i have seen Mesos used for managing big data frameworks like Spark. Any large distributed computing framework has to deal with some of the same problems - managing resources on worker nodes, sending tasks, getting results, scheduling etc. and Mesos/YARN provide a common base here, better than each framework duplicating the same code and solving the same problems.
  - StavrosK 12y ago
    That makes sense, thanks. The workers are pretty independent from each other, too, so this setup would be more fitting for that.
- justrudd 12y ago
  You can use it for web apps/services (https://mesosphere.com/docs/getting-started/service-discovery/ https://mesosphere.com/docs/getting-started/service-discover...). But I'm with you on databases. I guess I just like knowing that those hosts are being managed and cared for (pets vs. cattle). With that said, I often wonder how many people are using Mesos/Marathon before they have any need for it? Using it on for 4 hosts vs. 40 or 400?
  - tekacs 12y ago
    There's nothing wrong with Mesos on 4 hosts, though... Anything more than a single host requires additional co-ordination - picking Mesos is mostly no different than rolling alternate methods for doing so. It's not choice with the least overhead, but it's a possibility, for those for whom it has the right conveniences...
lobster_johnson 12y ago
I've looked at Mesos briefly, but it seems completely JVM-based, and that you're not prepared to build your whole stack around the current Java stack (Hadoop, Spark, Storm, Kafka, Zookeeper etc.), Mesos has zero utility. Is this accurate? As someone working on scaling microservices, I keep being disappointed by potentially useful services that turn out to require a JVM-based language such as Java or Scala. For example, Kafka looks very decent, but the high-level client is written in Java; if you're not on the JVM, you're stuck implementing a lot of the client yourself. As far as I know (from the last time I looked at this stuff), the Zookeeper client is similar, whereas Spark and Storm both require that you write processing code on the JVM, and libhdfs is apparently still JNI-based, not native. For someone using Docker, is there anything competing with Mesos that isn't wedded to Javaland?
- senex 12y ago
  There are frameworks that run on Mesos capable of running Docker containers. Check out Singularity, for example.
  - CoffeeDregs 12y ago
    AFAIK, the Docker capability is largely due to Mesos. Frameworks need the ability to specify a container type so that they can specify that a Docker container can run. But running a Docker container is the responsibility of Mesos. So Marathon and Chronos, both frameworks, are able to specify a job which uses a Docker container, while Mesos is able to run the job which uses a Docker container.
- CoffeeDregs 12y ago
  > [Not using Java -> ] Mesos has zero utility. Is this accurate? Not at all. My company has a cluster with a few hundred Slaves and we're mostly a Python shop, with some C++ for machine vision. Mesos certainly has its issues (as does everything), but its awfully nice for micro-services: if you can package your service into a Docker container, then you can launch it into the cluster and Chronos/Marathon/Mesos will take care of making sure that it's run/running. I missed the deadline for the Mesos conference (I was unaware of the deadline), but I'm trying to squeak in a talk about "Using Mesos at [small] Scale" because we're a small company and Mesos has allowed us to do a bunch of big company stuff. >is there anything competing with Mesos that isn't wedded to Javaland? Yes, I can look at the source code and see that it uses te JVM (Chronos uses Scala), but, AFAICT, Mesos isn't "wedded" to anything. All of the components are API-driven. I apt-get install it, I run it, I send jobs to it, it works and it behaves well. Better I can poke at the APIs of any of the services to find out what is happening. So we use Marathon for service-discovery and run Chronos, a framework, under Marathon. Makes finding Chronos, which could be on one of 200 machines, quick and easy.
CoffeeDregs 12y ago
As noted in another comment, we, a growing starting with 5 people around software out of 15 people total, run a Mesos cluster with a couple of hundred machines. By FAR our largest challenge has been to adapt our thinking to break the "this machine is 'production', that machine is 'staging'" mindset. We have a production compute infrastructure and you're welcome to launch 'production', 'staging' or whatever jobs into it. The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi... We have a separate, much smaller cluster to test new Mesos (and Chronos and Marathon) versions. But the distinctions "production", "staging" and "dev" have become much more nuanced, so we've settled on discussing the "application" environment versus the "infrastructure" environment. Much as, as a startup on AWS, you wouldn't distinguish the RDS instance of your database (e.g. 9.3.1), you would distinguish the version of your database on an RDS instance, we distinguish the versions of our apps on the production cluster and not the version of the production cluster. One of the team was an ex-Googler and he said that Google did much the same. The one thing Marathon and Chronos currently lack is a prioritization mechanism so we're building that as a Chronos task that monitors and scales down/up Marathon tasks by their priority (as represented in their id or tag).
- KMag 12y ago
  > The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi... Is there some common phrase relating belching and being happy that I haven't heard?
  - CoffeeDregs 12y ago
    Not that I know of, but it'd be awesome to have a word for joyous belching...

145 points by aant 12y ago

preillyme 12y ago

I'm curious which of the Marathon bugs caused your downtime?

aant 12y ago
I'm afraid I don't remember exactly which one it was but after an upgrade of the framework it worked perfectly again.
- florianleibert 12y ago
  The 0.7X releases contained a number of bugs. 0.8X has been significantly better!
phildougherty 12y ago
Don't poll the apps API if you have a decent number of tasks or it can bring the whole system to its knees.

wmf 12y ago

Maybe this is a good place to ask a question that I've been pondering for a while: Why is Mesos based on the concept of resource offers? AFAIK this is backwards compared to other schedulers where you ask for resources and the schedule gives them to you (or not).

aant 12y ago
I can't quite tell you why it was implemented like this. The Omega paper by Google describes different kinds of scheduling algorithms and their pros and cons. I don't think that it's backwards compared to other schedulers (many of them are monolithic) but I think it can be improved by using transactions and optimistic concurrency control. This is discussed in the Omega paper and I think there's work to integrate the shared state scheduler into Mesos.
sabraham 12y ago
To load the framework (application) with the logic with the concern of deciding what resources it wants. The scheduler shouldn't care about what you get, just fairness. Two-tiered scheduling achieves this: Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. http://mesos.berkeley.edu/mesos_tech_report.pdf http://mesos.berkeley.edu/mesos_tech_report.pdf
- wmf 12y ago
  I guess that makes sense if frameworks care about placement (which I generally don't) because otherwise they'd have to express placement constraints to Mesos. I still can't find documentation about what fairness policies are actually implemented, though.
  - sabraham 12y ago
    Mesos uses dominant resource fairness: https://www.cs.berkeley.edu/~alig/papers/drf.pdf https://www.cs.berkeley.edu/~alig/papers/drf.pdf
otterley 12y ago
The most puzzling thing for me is why schedulers and executors are so tightly coupled in Mesos. In my mind, those are separate concerns: schedulers should concern themselves solely with locating an available resource for you (and should be part of the Mesos core), and the executor should take care of launching and managing the program to be run. With the current design, if I want to launch a Docker container, I have to use its scheduler, even though its scheduler might not be as good as some other resource's scheduler. And I might have to accept a regression in the scheduler in order to benefit from a bugfix in the executor.

domenp 12y ago

Allow me take the opportunity and ask if anybody here is running Elasticsearch with Mesos (and Marathon) using Docker container? I'm running Elasticsearch nodes on a dedicated Mesos slaves and I’m still not sure how much memory should I allocate to Elasticsearch task e.g. all available memory as reported by Mesos or some smaller amount to leave something for the system? Please note that I'm asking about memory allocated to the Marathon task not about JVM heap size.

nemothekid 12y ago
I've taken the view of treating our database slaves as special and allocating all the memory on the slave to ES. That way ES (or any other db) always runs on the right slave, and no other potentially long running/resource hungry process runs alongside it (but just enough that's jenkins tasks and chronos tasks can still rune if need be). AKAIK, there isn't a right way to run databases because the persistent storage layer into mesos is still being baked.
- dingdingdang 12y ago
  > the persistent storage layer into mesos is still being baked That's pretty important point, would have thought persistent storage would have been first point to get working for a project like Mesos. Also, their homepage ( http://mesos.apache.org/ http://mesos.apache.org/ ) outright states: "Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines [...]". What storage are they talking about if not persistent storage?
  - nemothekid 12y ago
    This mesos 0.22 talk goes into a bit more what they are imagining - http://mesosphere.com/2015/03/27/mesos-0-22-0-released/ http://mesosphere.com/2015/03/27/mesos-0-22-0-released/ Essentially the current mesos disk quotas are just so that tasks that run, run with a minimum amount of disk space. I think what they are trying to accomplish in future releases is some way you could build an EBS Style disk space management on top of mesos.

twoy 12y ago

If you don't want to manage ZooKeeper manually to bootstrap Mesos cluster, you need another Mesos cluster haha.

saryant 12y ago
You could run it on top of something like CoreOS, but then you're just swapping Zookeeper for etcd.
eropple 12y ago
Or just use Exhibitor. I don't remember the last time I manually anything'd with Zookeeper. Between Exhibitor and Curator, I honestly find Zookeeper so straightforward and easy to work with that I don't quite understand the popularity of etcd.
- toomuchtodo 12y ago
  I'm running a small Exhibitor/Zookeeper cluster dockerized (5 nodes, several hundred clients), and its extremely straightforward. Etcd isn't what we'd consider production-grade yet.
  - saryant 12y ago
    Having worked with etcd in production for the last few months, I have to agree. The CoreOS stack needs some more time to marinate.
    - toomuchtodo 12y ago
      Thanks for this comment. Glad to know I made the right choice. I'm not saying etcd won't ever overshadow Zookeeper, it probably will with the momentum behind it, but as an ops guys, I wasn't willing to bet production application service discovery on it.
      - eropple 12y ago
        My distaste for the Go community is pretty well-established in these parts; I think worse-is-better is screwing us all, and etcd seems to me to be the worse-is-better Zookeeper. And for things that don't matter, sure, worse-is-better your life away; a Rails app can be whatever you want, but the infrastructure I manage had better be bulletproof. I won't say etcd will never be competitive, but without some significant changes, I don't see it getting my vote--and those changes are largely around the parts of the feature set that etcd doesn't support, at which point...why use it, anyway?

rckrd 12y ago

An interesting talk about Docker and Mesos by a former coworker, he also contributed to 0.22: https://www.youtube.com/watch?v=ZZXtXLvTXAE https://www.youtube.com/watch?v=ZZXtXLvTXAE

andyidsinga 12y ago

question / thought : seems like mesos could evolve to be an alternative to openstack ...especially if a tenant layer is developed ..yes/no ???

wmf 12y ago
Probably. And OpenShift 3.0 (also on the front page today) is a PaaS (Cloud Foundry competitor) built on Mesos.
- josephjacks 12y ago
  OpenShift is most definitely not built on Mesos. RedHat's plans over time are to integrate with Mesos at a lower level for fine grained resource allocation, but for now Kubernetes is used as the cluster scheduling and orchestrations runtime.

StavrosK 12y ago

Okay, maybe I'm getting old, but I can't really see the point of Mesos. It feels like it's adding too much magic to the mix, and I can't see myself using it to deploy web app servers/databases. Is there a different intended purpose that I'm missing? I suspect it's great for deploying task workers, for example.

bonobo3000 12y ago
Yeah i have seen Mesos used for managing big data frameworks like Spark. Any large distributed computing framework has to deal with some of the same problems - managing resources on worker nodes, sending tasks, getting results, scheduling etc. and Mesos/YARN provide a common base here, better than each framework duplicating the same code and solving the same problems.
- StavrosK 12y ago
  That makes sense, thanks. The workers are pretty independent from each other, too, so this setup would be more fitting for that.
justrudd 12y ago
You can use it for web apps/services (https://mesosphere.com/docs/getting-started/service-discovery/ https://mesosphere.com/docs/getting-started/service-discover...). But I'm with you on databases. I guess I just like knowing that those hosts are being managed and cared for (pets vs. cattle). With that said, I often wonder how many people are using Mesos/Marathon before they have any need for it? Using it on for 4 hosts vs. 40 or 400?
- tekacs 12y ago
  There's nothing wrong with Mesos on 4 hosts, though... Anything more than a single host requires additional co-ordination - picking Mesos is mostly no different than rolling alternate methods for doing so. It's not choice with the least overhead, but it's a possibility, for those for whom it has the right conveniences...

lobster_johnson 12y ago

I've looked at Mesos briefly, but it seems completely JVM-based, and that you're not prepared to build your whole stack around the current Java stack (Hadoop, Spark, Storm, Kafka, Zookeeper etc.), Mesos has zero utility. Is this accurate? As someone working on scaling microservices, I keep being disappointed by potentially useful services that turn out to require a JVM-based language such as Java or Scala. For example, Kafka looks very decent, but the high-level client is written in Java; if you're not on the JVM, you're stuck implementing a lot of the client yourself. As far as I know (from the last time I looked at this stuff), the Zookeeper client is similar, whereas Spark and Storm both require that you write processing code on the JVM, and libhdfs is apparently still JNI-based, not native. For someone using Docker, is there anything competing with Mesos that isn't wedded to Javaland?

senex 12y ago
There are frameworks that run on Mesos capable of running Docker containers. Check out Singularity, for example.
- CoffeeDregs 12y ago
  AFAIK, the Docker capability is largely due to Mesos. Frameworks need the ability to specify a container type so that they can specify that a Docker container can run. But running a Docker container is the responsibility of Mesos. So Marathon and Chronos, both frameworks, are able to specify a job which uses a Docker container, while Mesos is able to run the job which uses a Docker container.
CoffeeDregs 12y ago
> [Not using Java -> ] Mesos has zero utility. Is this accurate? Not at all. My company has a cluster with a few hundred Slaves and we're mostly a Python shop, with some C++ for machine vision. Mesos certainly has its issues (as does everything), but its awfully nice for micro-services: if you can package your service into a Docker container, then you can launch it into the cluster and Chronos/Marathon/Mesos will take care of making sure that it's run/running. I missed the deadline for the Mesos conference (I was unaware of the deadline), but I'm trying to squeak in a talk about "Using Mesos at [small] Scale" because we're a small company and Mesos has allowed us to do a bunch of big company stuff. >is there anything competing with Mesos that isn't wedded to Javaland? Yes, I can look at the source code and see that it uses te JVM (Chronos uses Scala), but, AFAICT, Mesos isn't "wedded" to anything. All of the components are API-driven. I apt-get install it, I run it, I send jobs to it, it works and it behaves well. Better I can poke at the APIs of any of the services to find out what is happening. So we use Marathon for service-discovery and run Chronos, a framework, under Marathon. Makes finding Chronos, which could be on one of 200 machines, quick and easy.

CoffeeDregs 12y ago

As noted in another comment, we, a growing starting with 5 people around software out of 15 people total, run a Mesos cluster with a couple of hundred machines. By FAR our largest challenge has been to adapt our thinking to break the "this machine is 'production', that machine is 'staging'" mindset. We have a production compute infrastructure and you're welcome to launch 'production', 'staging' or whatever jobs into it. The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi... We have a separate, much smaller cluster to test new Mesos (and Chronos and Marathon) versions. But the distinctions "production", "staging" and "dev" have become much more nuanced, so we've settled on discussing the "application" environment versus the "infrastructure" environment. Much as, as a startup on AWS, you wouldn't distinguish the RDS instance of your database (e.g. 9.3.1), you would distinguish the version of your database on an RDS instance, we distinguish the versions of our apps on the production cluster and not the version of the production cluster. One of the team was an ex-Googler and he said that Google did much the same. The one thing Marathon and Chronos currently lack is a prioritization mechanism so we're building that as a Chronos task that monitors and scales down/up Marathon tasks by their priority (as represented in their id or tag).

KMag 12y ago
> The friction around "running Mesos" has mostly been the friction of the air from exhalatoins of joy buffeting our esophagi... Is there some common phrase relating belching and being happy that I haven't heard?
- CoffeeDregs 12y ago
  Not that I know of, but it'd be awesome to have a word for joyous belching...