> I also don't agree with the author's assertion that Spark is "Scala centric". Yes, Spark is written in Scala, but PySpark is definitely a first class citizen.

> I also don't agree with the author's assertion that Spark is "Scala centric". Yes, Spark is written in Scala, but PySpark is definitely a first class citizen. To be fair, prior to Spark Dataframes (i.e. the days of pure RDDs), the only way to get performance out of Spark was to write Scala code. The serialization overhead of PySpark precluded it from large-scale data engineering workloads. Most companies rewrote their PySpark code in Scala for production. Now that we have Spark dataframes, PySpark performance is mostly on-par with ScalaSpark for many SQL-amenable operations. And with Apache Arrow in-memory support on Spark >2.3, the Python serialization overhead problem goes away. But Spark is still to some extent Scala-centric. The documentation is trilingual, but there is still a distinct Scala-first culture.