3 ms·
If you think these tools aren't necessary, you probably haven't managed a cluster with thousands of machines and hundreds of users. I imagine that every company
by necubi 13y ago
If you think these tools aren't necessary, you probably haven't managed a cluster with thousands of machines and hundreds of users. I imagine that every company in that situation has an ad-hoc implementation of distributed cron and distributed upstart (I know we do).
Without something like Mesos, you generally will run different things on your cluster by statically partitioning it (these ten racks run Hadoop, this rack runs our website, this rack runs Spark because some engineers wanted to try that out, etc.) or by running everything together (typically done with your distributed file system, but can be problematic with more compute-oriented services).
The mesos approach is to stop thinking of your cluster on a machine-by-machine or rack-by-rack basis, but instead as just a giant pool of resources. It's a very powerful abstraction that greatly increases the number of machines and developers that are manageable.
- peterwwillis 13y agoI'm familiar with the concept behind it. My problem was with how they all seem to do the same things, and nobody yet has pointed this out; everyone just accepts the fact that they're mostly redundant and moves on. I've managed SSI clusters, MPI clusters, and clusters of dumb app servers of varying sizes (10 nodes to 10,000). If you really want just a giant pool of resources, you can do much worse than an SSI cluster, but nobody wants to spend time working on a hard problem, so instead we dick around with task-shuffling job-runners inside the components that were written by the hardcore programmers that work in the kernel. But I guess we do what we can with what we have... (I blame Linus's team for not merging openMosix when they had the chance!)