3 ms·

Really, it's mesos that "makes your datacenter run like Google's," while Marathon is a framework built on top of it that provides distributed service management

by necubi 13y ago

Really, it's mesos that "makes your datacenter run like Google's," while Marathon is a framework built on top of it that provides distributed service management (think distributed upstart). Mesos is in part based on ideas from Google's Borg cluster management system. The basic idea is to provide the abstraction of a single machine to developers, who can say things like: I want to run this program on 40 cores with 100GB of memory, and the system will find those resources across the cluster, create a cgroup with your resource limits, and start your services. It also provides various tools for writing distributed services, like an implementation of PAXOS.

peterwwillis 13y ago
But Mesos already provides distributed service management. Why is Marathon doing it too? And why do you need Chronos if Marathon performs the same functions?
- florianleibert 13y ago
  Chronos is the "cron" for the cluster. Marathon is the "upstart" for your cluster. You can start Chronos via Marathon.
  - peterwwillis 13y ago
    Yes. I get that. Except Marathon can do 'cron-like' things too. So can Mason. Mason manages cluster applications. So does Marathon. Mason can juggle task resources. So does Marathon. Mason is a framework. So is Marathon. Mason has a scheduler. So does Marathon. The only thing Chronos does, at all, is run jobs on a schedule, distributedly. Which Marathon and Mason can do too. So can someone please explain to me why you need all three?? -- Look, here, at the infographics in the middle of the page: https://github.com/mesosphere/marathon https://github.com/mesosphere/marathon What they're saying is, Marathon will move your jobs to a new server when one dies. Okay, cool. But Chronos can do that too. And Mesos can do that too! -- I'm pretty sure all these tools are a giant troll by Google to get its competitors to burn R&D time on reinventing tools that already exist and aren't necessary.
    - necubi 13y ago
      If you think these tools aren't necessary, you probably haven't managed a cluster with thousands of machines and hundreds of users. I imagine that every company in that situation has an ad-hoc implementation of distributed cron and distributed upstart (I know we do). Without something like Mesos, you generally will run different things on your cluster by statically partitioning it (these ten racks run Hadoop, this rack runs our website, this rack runs Spark because some engineers wanted to try that out, etc.) or by running everything together (typically done with your distributed file system, but can be problematic with more compute-oriented services). The mesos approach is to stop thinking of your cluster on a machine-by-machine or rack-by-rack basis, but instead as just a giant pool of resources. It's a very powerful abstraction that greatly increases the number of machines and developers that are manageable.
      - peterwwillis 13y ago
        I'm familiar with the concept behind it. My problem was with how they all seem to do the same things, and nobody yet has pointed this out; everyone just accepts the fact that they're mostly redundant and moves on. I've managed SSI clusters, MPI clusters, and clusters of dumb app servers of varying sizes (10 nodes to 10,000). If you really want just a giant pool of resources, you can do much worse than an SSI cluster, but nobody wants to spend time working on a hard problem, so instead we dick around with task-shuffling job-runners inside the components that were written by the hardcore programmers that work in the kernel. But I guess we do what we can with what we have... (I blame Linus's team for not merging openMosix when they had the chance!)
    - florianleibert 13y ago
      I'll outline the specific goals of both Marathon / Chronos to clear any confusion: Marathon: Execute a long running job in the cluster and make sure it keeps on running. You can specify resource requirements as well as how many instances of this job you want to run. Examples of jobs: Rails App, Jetty Service, JBoss Service. Chronos: You can specify a repeating & finite job based on a schedule or another job completing. You specify resource requirements. Examples of jobs: Mysqldump (e.g. daily dump prod DB), Hadoop Job (cascading, pig, cascalog, scalding...), bash script using ImageMagick to create thumbnails. Both of these systems are Mesos frameworks - Mesos does the heavy lifting and offers resources to these frameworks which the frameworks can accept or reject. You can find more info here: http://mesos.apache.org/ http://mesos.apache.org/
  - florianleibert 13y ago
    Mesos cannot do "cron-like" things. Mesos is a resource manager for your cluster. You need a framework to actually schedule tasks. Marathon / Chronos are both frameworks.

by necubi 13y ago

peterwwillis 13y ago

But Mesos already provides distributed service management. Why is Marathon doing it too? And why do you need Chronos if Marathon performs the same functions?

florianleibert 13y ago
Chronos is the "cron" for the cluster. Marathon is the "upstart" for your cluster. You can start Chronos via Marathon.
- peterwwillis 13y ago
  Yes. I get that. Except Marathon can do 'cron-like' things too. So can Mason. Mason manages cluster applications. So does Marathon. Mason can juggle task resources. So does Marathon. Mason is a framework. So is Marathon. Mason has a scheduler. So does Marathon. The only thing Chronos does, at all, is run jobs on a schedule, distributedly. Which Marathon and Mason can do too. So can someone please explain to me why you need all three?? -- Look, here, at the infographics in the middle of the page: https://github.com/mesosphere/marathon https://github.com/mesosphere/marathon What they're saying is, Marathon will move your jobs to a new server when one dies. Okay, cool. But Chronos can do that too. And Mesos can do that too! -- I'm pretty sure all these tools are a giant troll by Google to get its competitors to burn R&D time on reinventing tools that already exist and aren't necessary.
  - necubi 13y ago
    If you think these tools aren't necessary, you probably haven't managed a cluster with thousands of machines and hundreds of users. I imagine that every company in that situation has an ad-hoc implementation of distributed cron and distributed upstart (I know we do). Without something like Mesos, you generally will run different things on your cluster by statically partitioning it (these ten racks run Hadoop, this rack runs our website, this rack runs Spark because some engineers wanted to try that out, etc.) or by running everything together (typically done with your distributed file system, but can be problematic with more compute-oriented services). The mesos approach is to stop thinking of your cluster on a machine-by-machine or rack-by-rack basis, but instead as just a giant pool of resources. It's a very powerful abstraction that greatly increases the number of machines and developers that are manageable.
    - peterwwillis 13y ago
      I'm familiar with the concept behind it. My problem was with how they all seem to do the same things, and nobody yet has pointed this out; everyone just accepts the fact that they're mostly redundant and moves on. I've managed SSI clusters, MPI clusters, and clusters of dumb app servers of varying sizes (10 nodes to 10,000). If you really want just a giant pool of resources, you can do much worse than an SSI cluster, but nobody wants to spend time working on a hard problem, so instead we dick around with task-shuffling job-runners inside the components that were written by the hardcore programmers that work in the kernel. But I guess we do what we can with what we have... (I blame Linus's team for not merging openMosix when they had the chance!)
  - florianleibert 13y ago
    I'll outline the specific goals of both Marathon / Chronos to clear any confusion: Marathon: Execute a long running job in the cluster and make sure it keeps on running. You can specify resource requirements as well as how many instances of this job you want to run. Examples of jobs: Rails App, Jetty Service, JBoss Service. Chronos: You can specify a repeating & finite job based on a schedule or another job completing. You specify resource requirements. Examples of jobs: Mysqldump (e.g. daily dump prod DB), Hadoop Job (cascading, pig, cascalog, scalding...), bash script using ImageMagick to create thumbnails. Both of these systems are Mesos frameworks - Mesos does the heavy lifting and offers resources to these frameworks which the frameworks can accept or reject. You can find more info here: http://mesos.apache.org/ http://mesos.apache.org/
- florianleibert 13y ago
  Mesos cannot do "cron-like" things. Mesos is a resource manager for your cluster. You need a framework to actually schedule tasks. Marathon / Chronos are both frameworks.