Missing from this conversation is the fact that Scala used to be the goto language for writing Spark applications, but with the new developments in Apache Arrow

Missing from this conversation is the fact that Scala used to be the goto language for writing Spark applications, but with the new developments in Apache Arrow, PySpark runs your big data stuff just as efficiently, therefore Python has replaced Scala as the lingua franca of big data.