3 ms·

At the beginning of the book it says 'The real power of Spark is that this query can be run on a laptop or on a cluster of hundreds of servers with no code chan

At the beginning of the book it says 'The real power of Spark is that this query can be run on a laptop or on a cluster of hundreds of servers with no code changes required.' Is this accurate? I thought the power of Spark is that data stays in memory in each node in the cluster. Being able to run on a cluster in a fault tolerant manner was achieved already by Hadoop ecosystem.

zjaffee 3y ago
The power of spark is truthfully that it's really good at running sort + aggregate type queries on columnar stored data. Once you start needing to do complicated joins it becomes more about how you structure everything.
adammarples 3y ago
It was achieved by hadoop first, but it is still a big feature of spark
Bootvis 3y ago
Except that doing very, very simple stuff, like counting 10 rows, on a laptop is excruciatingly slow.
- qsort 3y ago
  Well I mean, that's a problem with all those solutions. If your dataset is so small it fits in RAM it's simply ridiculous to reach for those tools, and you don't start seeing real advantages until you have truly massive datasets. Anecdotally using Spark on less than 1TB just means wasting your time, but in Spark's defense, that's not what it's for.
  - iamcreasy 3y ago
    Do you mean 1TB of data? or cluster with total memory capacity less than 1TB?
    - qsort 3y ago
      I mean data size. It's anecdotally the threshold where "normal" ETL methods start having problems. Obviously performance is hard and the only correct answer is "it depends", but I have never seen Spark improve things on datasets that small.
ramraj07 3y ago
In theory sure in practice I beg to differ. You tend to end up writing different levels of annoyingly intricate spark code as your data volume, processing comolexity (and infra constraints) increase. Anyone who says this is not true hasn't tried to join two 10 TB dataframes on a highly skewed key with a hundred million unique values without spending thousands of dollars and without failing at random instances.
poorman 3y ago
I think in general, the idea is that you can execute SQL using the query engine on Spark.

zjaffee 3y ago

The power of spark is truthfully that it's really good at running sort + aggregate type queries on columnar stored data. Once you start needing to do complicated joins it becomes more about how you structure everything.

adammarples 3y ago

It was achieved by hadoop first, but it is still a big feature of spark

Bootvis 3y ago

Except that doing very, very simple stuff, like counting 10 rows, on a laptop is excruciatingly slow.

qsort 3y ago
Well I mean, that's a problem with all those solutions. If your dataset is so small it fits in RAM it's simply ridiculous to reach for those tools, and you don't start seeing real advantages until you have truly massive datasets. Anecdotally using Spark on less than 1TB just means wasting your time, but in Spark's defense, that's not what it's for.
- iamcreasy 3y ago
  Do you mean 1TB of data? or cluster with total memory capacity less than 1TB?
  - qsort 3y ago
    I mean data size. It's anecdotally the threshold where "normal" ETL methods start having problems. Obviously performance is hard and the only correct answer is "it depends", but I have never seen Spark improve things on datasets that small.

ramraj07 3y ago

In theory sure in practice I beg to differ. You tend to end up writing different levels of annoyingly intricate spark code as your data volume, processing comolexity (and infra constraints) increase. Anyone who says this is not true hasn't tried to join two 10 TB dataframes on a highly skewed key with a hundred million unique values without spending thousands of dollars and without failing at random instances.

poorman 3y ago

I think in general, the idea is that you can execute SQL using the query engine on Spark.