3 ms·
"Fully managed Spark" sounded awesome a few years ago. But now, Spark can run on Kubernetes clusters [1]. If your infrastructure is already running on Kubernete
by flowerlad 6y ago
"Fully managed Spark" sounded awesome a few years ago. But now, Spark can run on Kubernetes clusters [1]. If your infrastructure is already running on Kubernetes then you already have a cluster capable of running Spark. And because of the magic of Kubernetes you don't even have to dedicate nodes to Spark.
[1] https://spark.apache.org/docs/latest/running-on-kubernetes.html https://spark.apache.org/docs/latest/running-on-kubernetes.h...
- bpodgursky 6y ago100%. The DataBricks notebooks have a lot of value (for now) but ever since running Spark on Kube... I have had literally 0 cluster issues. It's absolutely shocking, coming from a YARN-based environment, where I was constantly plagued with ResourceManager issues, autoscaling issues, preemption issues, network disconnect issues...