PySpark should only be used for prototyping. It add an enormous extra overhead on operations due to serializing data back and forth between the Java and Python

PySpark should only be used for prototyping. It add an enormous extra overhead on operations due to serializing data back and forth between the Java and Python processes.