A bit of the history: The Mesosphere DCOS is built around the Apache Mesos kernel The Mesos kernel was developed at UC Berkeley in 2009 [1]. Spark was writte

A bit of the history: The Mesosphere DCOS is built around the Apache Mesos kernel The Mesos kernel was developed at UC Berkeley in 2009 [1]. Spark was written as a sample app on top of it [2]. Ben Hindman and his colleagues at the UC Berkeley AmpLab had always envisioned Mesos as a kernel inside of a full-blown operating system [3]. They finally brought it to market. [1] https://www.usenix.org/legacy/event/nsdi11/tech/full_papers/Hindman.pdf https://www.usenix.org/legacy/event/nsdi11/tech/full_papers/... [2] "We have implemented Mesos in 10,000 lines of C++. The system scales to 50,000 (emulated) nodes and uses ZooKeeper for fault tolerance. To evaluate Mesos, we have ported three cluster computing systems to run over it: Hadoop, MPI, and the Torque batch scheduler. To validate our hypothesis that specialized frameworks provide value over general ones, we have also built a new framework on top of Mesos called Spark, optimized for iterative jobs where a dataset is reused in many parallel operations, and shown that Spark can outperform Hadoop by 10x in iterative machine learning workloads." ibid. [3] http://people.csail.mit.edu/matei/papers/2011/hotcloud_datacenter_os.pdf http://people.csail.mit.edu/matei/papers/2011/hotcloud_datac...