4 ms·

I've been personally very impressed with Spark's RDD api for easily parallelizing tasks that are "embarrassingly parallel". However, I have found the data frame

I've been personally very impressed with Spark's RDD api for easily parallelizing tasks that are "embarrassingly parallel". However, I have found the data frames API to not always work as advertised and thus I'm very skeptical of the benchmarks. I think a prime example of this is I was using some very basic windowing functions and due to the data shuffling (The data wasn't naturally partitioned) it seemed to be very buggy and not very clear why stuff was failing. I ended up rewriting the same section of code using hive and it had both better performance and didn't seem to have any odd failures. I realize this stuff will improve but I'm still skeptical.

tma-1 10y ago
I have been extensively using the dateframe/sql API and I just love it. Most of the issues I have had stemmed from the cluster / Spark configuration and not the API itself. Using SQL is so much more intuitive them using multiple joins, selects, filter etc on an rdd.
- mastratton3 10y ago
  So I did find it useful for doing additional exploratory aggregations once the data was already cleaned and denormalized. My comment was more directed at the upfront initial data processing (In our case, extracting time series data out of a large amount of files). I did hit issues w/ multiple joins and shuffling though. Have you not hit issues w/ shuffling? I was using Spark 1.5.1 for the record.
  - tma-1 10y ago
    Have you tried tuning Spark's memory parameters?
SixSigma 10y ago
How difficult was the bug reporting process? That is a good metric for a project.
- threeseed 10y ago
  Bug reporting is simple as per any Apache project i.e. JIRA. But let's be clear here Spark is not a 'real' open source project. It's a dictatorship run by Databricks (in the nicest possible way). There are 414 outstanding pull requests some of which came from the likes of Intel which added subquery support over a year ago. Never got merged. The best metric for any open source project is how easily can you contribute and improve the product.
  - kod 10y ago
    I totally agree that's the best metric. I totally disagree with your characterization of Spark. I showed up on the mailing list, said "hey, here's some stuff that was useful for me, let me know if it's useful for anyone else." A committer said, "cool, send a PR". Couple of rounds of code review and it was merged. Later on, bigger features required a lot more discussion, but as long as I was willing to follow up, I got my changes in. Spark has on the order of 1,000 contributors. I'd call that a "real" open source project.
    - rxin 10y ago
      Please keep the contributions coming!

tma-1 10y ago

I have been extensively using the dateframe/sql API and I just love it. Most of the issues I have had stemmed from the cluster / Spark configuration and not the API itself. Using SQL is so much more intuitive them using multiple joins, selects, filter etc on an rdd.

mastratton3 10y ago
So I did find it useful for doing additional exploratory aggregations once the data was already cleaned and denormalized. My comment was more directed at the upfront initial data processing (In our case, extracting time series data out of a large amount of files). I did hit issues w/ multiple joins and shuffling though. Have you not hit issues w/ shuffling? I was using Spark 1.5.1 for the record.
- tma-1 10y ago
  Have you tried tuning Spark's memory parameters?

SixSigma 10y ago

How difficult was the bug reporting process? That is a good metric for a project.

threeseed 10y ago
Bug reporting is simple as per any Apache project i.e. JIRA. But let's be clear here Spark is not a 'real' open source project. It's a dictatorship run by Databricks (in the nicest possible way). There are 414 outstanding pull requests some of which came from the likes of Intel which added subquery support over a year ago. Never got merged. The best metric for any open source project is how easily can you contribute and improve the product.
- kod 10y ago
  I totally agree that's the best metric. I totally disagree with your characterization of Spark. I showed up on the mailing list, said "hey, here's some stuff that was useful for me, let me know if it's useful for anyone else." A committer said, "cool, send a PR". Couple of rounds of code review and it was merged. Later on, bigger features required a lot more discussion, but as long as I was willing to follow up, I got my changes in. Spark has on the order of 1,000 contributors. I'd call that a "real" open source project.
  - rxin 10y ago
    Please keep the contributions coming!