4 ms·
I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs. I migrated to 2.0 a few months ago and have be
by ap22213 10y ago
I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs.
I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on.
I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface.
However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.
- mziel 10y agoDo you have a write-up of your experience by any chance? We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.
- sandGorgon 10y agoWe don't do a lot of ML processing..so my experience is very limited. Do you see yourself getting better performance with the Dataframes+ Pipelines infra...or is it simply a better mental model.
- deleted 10y ago[deleted]
- rxin 10y agoReynold from Databricks and the Apache Spark project here. Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?
- snnn 10y agoHi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
- sandGorgon 10y agoOh wow... I didn't even know these limitations (and in fact ,they are not very Google-able). Can you talk about these limitations and your experience?
- snnn 10y agohttps://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-6235 https://issues.apache.org/jira/browse/SPARK-6235 You'll hit this bug when your model size is larger than 2GB. BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that. https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-5490 https://issues.apache.org/jira/browse/SPARK-5490