3 ms·

I have no idea what any of the google tech has to do with anything here. Quoting from the original spark paper: > Spark is built on top of Mesos [16, 15], a “

• dekhn 2y ago
  The RAD lab folks who built Mesos were aware of Borg and how it approached the problem of schedling a bunch of different jobs on a collection of disparate hardware. Prior to borg, most large-scale clusters were managed with batch queue software, while borg and mesos are more from the "service management"- a collection of jobs that run concurrently, with priority levels used to preempt lower-priority jobs to allow higher-priority jobs to schedule and run "immediately". The need for this pops up for nearly every large scale data processing enterprise- with k8s replacing mesos, yarn, and other systems as the cluster scheduler du-jour. One of the big advantages of a service scheduler versus a batch queue is that you can implement a batch queue on top of a service scheduler much more easily than you can implement a service scheduler on top of a batch queue.