4 ms·

The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model. Much nicer IMHO https://databricks.com/blog/2016/

The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model. Much nicer IMHO https://databricks.com/blog/2016/03/03/introducing-graphframes.html https://databricks.com/blog/2016/03/03/introducing-graphfram...

mziel 10y ago
GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.
taeric 10y ago
I'm curious just how beneficial a giant all inclusive framework is for this. My gut is it will be a huge boon for most of us, but you will still find very valid use cases for rolling your own. That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.
ap22213 10y ago
I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs. I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on. I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface. However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.
- mziel 10y ago
  Do you have a write-up of your experience by any chance? We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.
  - sandGorgon 10y ago
    We don't do a lot of ML processing..so my experience is very limited. Do you see yourself getting better performance with the Dataframes+ Pipelines infra...or is it simply a better mental model.
- deleted 10y ago
  [deleted]
- rxin 10y ago
  Reynold from Databricks and the Apache Spark project here. Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?
  - snnn 10y ago
    Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
    - sandGorgon 10y ago
      Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able). Can you talk about these limitations and your experience?
      - snnn 10y ago
        https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-6235 https://issues.apache.org/jira/browse/SPARK-6235 You'll hit this bug when your model size is larger than 2GB. BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that. https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-5490 https://issues.apache.org/jira/browse/SPARK-5490

mziel 10y ago

GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.

taeric 10y ago

I'm curious just how beneficial a giant all inclusive framework is for this. My gut is it will be a huge boon for most of us, but you will still find very valid use cases for rolling your own. That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.

ap22213 10y ago

I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs. I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on. I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface. However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.

mziel 10y ago
Do you have a write-up of your experience by any chance? We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.
- sandGorgon 10y ago
  We don't do a lot of ML processing..so my experience is very limited. Do you see yourself getting better performance with the Dataframes+ Pipelines infra...or is it simply a better mental model.
deleted 10y ago
[deleted]
rxin 10y ago
Reynold from Databricks and the Apache Spark project here. Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?
- snnn 10y ago
  Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
  - sandGorgon 10y ago
    Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able). Can you talk about these limitations and your experience?
    - snnn 10y ago
      https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-6235 https://issues.apache.org/jira/browse/SPARK-6235 You'll hit this bug when your model size is larger than 2GB. BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that. https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-5490 https://issues.apache.org/jira/browse/SPARK-5490