3 ms·
To be honest PySpark and SparkR are always going to be 2nd category citizens (because of the serialization/pickling between the two environments). Databricks sh
by mziel 11y ago
To be honest PySpark and SparkR are always going to be 2nd category citizens (because of the serialization/pickling between the two environments). Databricks shows nice graphs, saying they are equivalent for DataFrames, however those count only for built-in functions that basically translate code into execution plan for Catalyst. For anything bespoke (UDFs, custom Transformers/Estimators) you're better off using Scala.
- rxin 11y agoThis is true when you compare the performance vs Java/Scala, but if you compare it with other tools that are native in Python, it is not really much worse. For examples, Pandas operations that use custom UDFs are substantially slower than the native operations. That said, as part of Project Tungsten, we have some ideas about a batch columnar format that can be shared by Python, R, Scala and Java, and that should be able to eliminate most of the inefficiency in serialization across process boundaries.
- mziel 11y agoThat sounds very interesting. Is there any ticket, where I can follow the progress on the batch columnar format you mentioned? Btw, I was critical about the issue above, but I do love Spark, using it on a daily basis. :)
- rxin 11y agoI just created a JIRA ticket tracking this: https://issues.apache.org/jira/browse/SPARK-12635 https://issues.apache.org/jira/browse/SPARK-12635 Thanks for the reminder!
- minimaxir 11y agoThat's fair; mostly I'd prefer keeping everything in Python due to the other data implications outside of Spark (e.g bespoke data cleanup with Python syntax) and I'd prefer to keep the stack smaller.
- xjlin0 11y agoYes that's right. Team: could you make more API available through PySPark, please?