3 ms·

So the detailed post on datasets at https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html https://databricks.com/blog/2016/01/04/introducing-sp

by kod 11y ago

So the detailed post on datasets at https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html https://databricks.com/blog/2016/01/04/introducing-spark-dat... uses groupBy I'm pretty sure based on previous comments you've made that groupBy was one of the things you'd rather eliminate from the RDD api, because of the performance impact compared to reduceByKey (which is almost always what people should be using instead). Are you at all worried about confusion if groupBy now performs ok on datasets, but not on rdds?

rxin 11y ago
Despite our attempts at warning people, a lot of users still use groupByKey in RDDs. Hopefully over time this won't be a problem as the engine should be able to figure out more intelligently and do the proper rewrite (of course, we won't be able to do it 100%).
- gshayban 11y ago
  Many people blindly point to the docs to say "don't use groupBy, prefer reduce because it's faster..." Are there better examples that illustrate the fundamental differences between the two operations? Surely there is still a need for both operations
  - IvanVergiliev 11y ago
    Reduce can perform reductions on locally on each machine before shuffling the data. This decreases the memory as well as the network overhead. If you need all the elements for a given key - e.g. to display them to a user or save them to a DB, perhaps you should use groupBy. If you're going to perform some form of a reduce after that though, it's likely sub-optimal.
  - dastbe 11y ago
    databricks has a page that describes the pitfalls: https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html https://databricks.gitbooks.io/databricks-spark-knowledge-ba... I don't know if the OutOfMemory exception can still occur in recent versions of Spark, but the performance impact of groupByKey is very real.
deleted 11y ago
[deleted]

by kod 11y ago

rxin 11y ago

Despite our attempts at warning people, a lot of users still use groupByKey in RDDs. Hopefully over time this won't be a problem as the engine should be able to figure out more intelligently and do the proper rewrite (of course, we won't be able to do it 100%).

gshayban 11y ago
Many people blindly point to the docs to say "don't use groupBy, prefer reduce because it's faster..." Are there better examples that illustrate the fundamental differences between the two operations? Surely there is still a need for both operations
- IvanVergiliev 11y ago
  Reduce can perform reductions on locally on each machine before shuffling the data. This decreases the memory as well as the network overhead. If you need all the elements for a given key - e.g. to display them to a user or save them to a DB, perhaps you should use groupBy. If you're going to perform some form of a reduce after that though, it's likely sub-optimal.
- dastbe 11y ago
  databricks has a page that describes the pitfalls: https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html https://databricks.gitbooks.io/databricks-spark-knowledge-ba... I don't know if the OutOfMemory exception can still occur in recent versions of Spark, but the performance impact of groupByKey is very real.

deleted 11y ago

[deleted]