3 ms·

This article makes it sound like Spark needs to be run on Hadoop clusters and that's not the case. Spark can be run on object stores like AWS S3 and Azure. I

This article makes it sound like Spark needs to be run on Hadoop clusters and that's not the case. Spark can be run on object stores like AWS S3 and Azure. I also don't agree with the author's assertion that Spark is "Scala centric". Yes, Spark is written in Scala, but PySpark is definitely a first class citizen. Databricks maintains a MLFlow project to make it easy to use Python with Spark: https://databricks.com/blog/2018/06/05/introducing-mlflow-an-open-source-machine-learning-platform.html https://databricks.com/blog/2018/06/05/introducing-mlflow-an...

wenc 8y ago
> I also don't agree with the author's assertion that Spark is "Scala centric". Yes, Spark is written in Scala, but PySpark is definitely a first class citizen. To be fair, prior to Spark Dataframes (i.e. the days of pure RDDs), the only way to get performance out of Spark was to write Scala code. The serialization overhead of PySpark precluded it from large-scale data engineering workloads. Most companies rewrote their PySpark code in Scala for production. Now that we have Spark dataframes, PySpark performance is mostly on-par with ScalaSpark for many SQL-amenable operations. And with Apache Arrow in-memory support on Spark >2.3, the Python serialization overhead problem goes away. But Spark is still to some extent Scala-centric. The documentation is trilingual, but there is still a distinct Scala-first culture.