3 ms·
FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probabl
by bunderbunder 8y ago
FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.
- dralley 8y agoYes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though. In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
- fs111 8y agospark clusters still have a spof as far as I understand the architecture.
- bunderbunder 8y agoTwo of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver. I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
- shub 8y agoIf you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA. You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.