3 ms·

I'll give it a shot, with the disclaimer that this explanation is based on conference talks I've seen and documentation I've read. Spark is a successor to Hado

by bjt 12y ago

I'll give it a shot, with the disclaimer that this explanation is based on conference talks I've seen and documentation I've read. Spark is a successor to Hadoop, aiming to perform big distributed data crunching jobs more quickly by not limiting itself to a map-reduce paradigm and by holding more data in memory. Apache Mesos is a resource scheduler for a cluster of machines. It is architected to be agnostic about the kind of application running on it, but I get the impression it's primarily used to make things like Hadoop and Spark run on the same cluster without stepping on each other's toes (and not used so much for long-running services like a web app). An application says to Mesos "hey I'd like to run job X", outlining the resources that it needs, and then Mesos looks at the cluster to see the best place to run the job. Mesos has some Docker integration but it's weird. Instead of launching your job and creating cgroups directly as it usually does, Mesos has to run a proxy process that talks to Docker and asks it to do it. (This is because Docker no longer has standalone mode. Which is my biggest gripe about Docker.) Kubernetes is solving a similar problem to Mesos, how to allocate resources in a cluster to all the jobs that you might want to run on it. It's explicitly focused on "containers", though not necessarily just Docker containers. (The README at https://github.com/GoogleCloudPlatform/kubernetes https://github.com/GoogleCloudPlatform/kubernetes makes no mention of Docker.) I get the impression it's focused more on long-running processes like web servers and less on ephemeral Hadoop-style jobs.

vertex-four 12y ago
> and not used so much for long-running services like a web app Mesosphere Marathon, which sits on top of Mesos, makes doing long-running services easier; it starts services, restarts them if machines crash, etc etc. Where Mesos is the resource schedular, Marathon is more like a *nix init system. Aurora is supposed to do much the same thing, but is in Incubation at the moment. There's also "Singularity", which does a bunch of stuff including managing long-running services and one-off services through an HTTP API and webapp. Mesos is supposed to be treated sort of as a "cluster kernel", with "frameworks" which sit on top of it and use it to schedule things. It's a lot more versatile than "I want to deploy a bunch of services" - it's more along the lines of having your own EC2 that you can request resources from on-demand.
- dln_eintr 12y ago
  Kubernetes also fits very well with Mesos for scheduling. Check out https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos

by bjt 12y ago

vertex-four 12y ago

> and not used so much for long-running services like a web app Mesosphere Marathon, which sits on top of Mesos, makes doing long-running services easier; it starts services, restarts them if machines crash, etc etc. Where Mesos is the resource schedular, Marathon is more like a *nix init system. Aurora is supposed to do much the same thing, but is in Incubation at the moment. There's also "Singularity", which does a bunch of stuff including managing long-running services and one-off services through an HTTP API and webapp. Mesos is supposed to be treated sort of as a "cluster kernel", with "frameworks" which sit on top of it and use it to schedule things. It's a lot more versatile than "I want to deploy a bunch of services" - it's more along the lines of having your own EC2 that you can request resources from on-demand.

dln_eintr 12y ago
Kubernetes also fits very well with Mesos for scheduling. Check out https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos