3 ms·
They are faster in the sense that many thing you previously had to do in a RDD lambda are now Dataframe operations (which are optimized by the Catalyst compiler
by nl 10y ago
They are faster in the sense that many thing you previously had to do in a RDD lambda are now Dataframe operations (which are optimized by the Catalyst compiler).
So if you want to do one of the operations in the sql.functions package[1] then Dataframes (and Datasets) are very valuable.
If not, then they won't give you much benefit. However, you will get a little improvement because the Tachyon out-of-JVM-memory framework[2] which I don't think RDD version has access to.
[1] http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#module-pyspark.sql.functions http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.ht...
[2] https://dzone.com/articles/Accelerate-In-Memory-Processing-with-Spark-from-Hours-to-Seconds-With-Tachyon https://dzone.com/articles/Accelerate-In-Memory-Processing-w...