4 ms·

Can someone explain why the in-memory caching is such a big win? Does Hadoop MapReduce not do caching as well? I'd expect at least filesystem caching when the

Can someone explain why the in-memory caching is such a big win? Does Hadoop MapReduce not do caching as well? I'd expect at least filesystem caching when the computation is running on the same machine as the data block...

mthomas 13y ago
My understanding of hadoop and hdfs is that both input and ourput are to/from disk. Your MR jobs are spun up and pointed to hdfs urls to read input from and they write to hdfs when done. This means that when there is an error, the intermediate computations don't necessarily have to be redone. However there is a trade off. Additionally, I believe that HDFS keeps 3 copies of the data around on 3 different nodes for redundancy. So there is the overhead of that network traffic.
- justinsb 13y ago
  I did find this (now-fixed) bug/enhancement: https://issues.apache.org/jira/browse/HDFS-2246 https://issues.apache.org/jira/browse/HDFS-2246 Sounds like, with that configuration applied, the in-memory performance difference between Hadoop and Spark should not be nearly as large.
rxin 13y ago
Spark goes much more beyond just in-memory caching. It features a more advanced scheduler and a higher level programming abstraction. The programming abstraction treats all data as collections (RDD in Spark terminology), and allow programmers to apply bulk transformation on these collections. Some examples of operations you can apply include the traditional map and reduce, the relational filter, join, outerJoin, and more advanced ones like sample. This abstraction makes it much easier to write distributed programs. As the Wired article mentioned, a distributed program written in Spark often looks identical to a single node program. This substantially reduces the amount of code one needs to write for distributed programs, and the best part is the code really expresses the algorithm (rather than cluttered with JobConf setup). And the scheduler and the engine itself are aware of the general DAG of operators, so they can schedule and run those operators better. For example, if you have multiple maps, the execution gets pipelined; if you are joining two collections that are partitioned the same way, the execution avoids an expensive shuffle step. There are many other benefits too. I'd encourage you to give a try. Thanks! Disclaimer: I am on the Spark team at UC Berkeley.
- justinsb 13y ago
  Thanks - a nice explanation of Spark's benefits. The programming & execution models sound like big improvements - I'll check it out!
- HCIdivision17 13y ago
  Does Spark address the same problems as Storm? It doesn't look like there's been any direct comparison of the two, though it looks like there's overlap. (I've wanted to start a streaming data processing project, and this looks like it would be good to consider for it.)
  - ikhahmedov 13y ago
    Last versions of Spark has ability to do stream processing. http://spark-project.org/docs/latest/streaming-programming-guide.html http://spark-project.org/docs/latest/streaming-programming-g...
  - krcz 13y ago
    There are some nice looking Spark vs Storm graphs in their streaming presentation slides: http://spark-project.org/talks/strata_spark_streaming.pdf http://spark-project.org/talks/strata_spark_streaming.pdf . Makes me wonder how biased these might be.
flatfilefan 13y ago
Unlike linear regression with normally distributed residuals, it is not possible to find a closed-form expression for the coefficient values that maximizes the likelihood function, so an iterative process must be used instead. Wikipedia
- joshhart 13y ago
  You can find closed-form expressions for SOME likelihood functions. This is the case for logistic regression. Anyway, the takeaway that I think you were downvoted for omitting is that if you can bypass the serialization, startup, and shutdown steps associated with the traditional Hadoop iterative process you'll get a huge speedup.
hobbyist 13y ago
Good question. I did read the spark paper, and one reason that I found for spark doing so much better than hadoop was that it avoids the unnecessary serialization, deserialization which hadoop just can not avoid. The RDD's as mentioned by @rxin, are in memory objects and thus do not require frequent serialization/deserialization when multiple operations are being applied to data.
joshhart 13y ago
Justin: 1. Hadoop is multi-tenant. Another job could burst your cache. Unless things have dramatically changed since I first read the source a year or two ago, spark jobs basically grab control of the cluster so that the cache won't get evicted. 2. Think of Pregel and other iterative-style jobs. You may run several iterations before writing to disk to checkpoint. This can speed up your job immensely since you don't have to write & read data from slow disks. Of course you lose on fault tolerance but the speedup may be worth it.

mthomas 13y ago

My understanding of hadoop and hdfs is that both input and ourput are to/from disk. Your MR jobs are spun up and pointed to hdfs urls to read input from and they write to hdfs when done. This means that when there is an error, the intermediate computations don't necessarily have to be redone. However there is a trade off. Additionally, I believe that HDFS keeps 3 copies of the data around on 3 different nodes for redundancy. So there is the overhead of that network traffic.

justinsb 13y ago
I did find this (now-fixed) bug/enhancement: https://issues.apache.org/jira/browse/HDFS-2246 https://issues.apache.org/jira/browse/HDFS-2246 Sounds like, with that configuration applied, the in-memory performance difference between Hadoop and Spark should not be nearly as large.

rxin 13y ago

Spark goes much more beyond just in-memory caching. It features a more advanced scheduler and a higher level programming abstraction. The programming abstraction treats all data as collections (RDD in Spark terminology), and allow programmers to apply bulk transformation on these collections. Some examples of operations you can apply include the traditional map and reduce, the relational filter, join, outerJoin, and more advanced ones like sample. This abstraction makes it much easier to write distributed programs. As the Wired article mentioned, a distributed program written in Spark often looks identical to a single node program. This substantially reduces the amount of code one needs to write for distributed programs, and the best part is the code really expresses the algorithm (rather than cluttered with JobConf setup). And the scheduler and the engine itself are aware of the general DAG of operators, so they can schedule and run those operators better. For example, if you have multiple maps, the execution gets pipelined; if you are joining two collections that are partitioned the same way, the execution avoids an expensive shuffle step. There are many other benefits too. I'd encourage you to give a try. Thanks! Disclaimer: I am on the Spark team at UC Berkeley.

justinsb 13y ago
Thanks - a nice explanation of Spark's benefits. The programming & execution models sound like big improvements - I'll check it out!
HCIdivision17 13y ago
Does Spark address the same problems as Storm? It doesn't look like there's been any direct comparison of the two, though it looks like there's overlap. (I've wanted to start a streaming data processing project, and this looks like it would be good to consider for it.)
- ikhahmedov 13y ago
  Last versions of Spark has ability to do stream processing. http://spark-project.org/docs/latest/streaming-programming-guide.html http://spark-project.org/docs/latest/streaming-programming-g...
- krcz 13y ago
  There are some nice looking Spark vs Storm graphs in their streaming presentation slides: http://spark-project.org/talks/strata_spark_streaming.pdf http://spark-project.org/talks/strata_spark_streaming.pdf . Makes me wonder how biased these might be.

flatfilefan 13y ago

Unlike linear regression with normally distributed residuals, it is not possible to find a closed-form expression for the coefficient values that maximizes the likelihood function, so an iterative process must be used instead. Wikipedia

joshhart 13y ago
You can find closed-form expressions for SOME likelihood functions. This is the case for logistic regression. Anyway, the takeaway that I think you were downvoted for omitting is that if you can bypass the serialization, startup, and shutdown steps associated with the traditional Hadoop iterative process you'll get a huge speedup.

hobbyist 13y ago

Good question. I did read the spark paper, and one reason that I found for spark doing so much better than hadoop was that it avoids the unnecessary serialization, deserialization which hadoop just can not avoid. The RDD's as mentioned by @rxin, are in memory objects and thus do not require frequent serialization/deserialization when multiple operations are being applied to data.

joshhart 13y ago

Justin: 1. Hadoop is multi-tenant. Another job could burst your cache. Unless things have dramatically changed since I first read the source a year or two ago, spark jobs basically grab control of the cluster so that the cache won't get evicted. 2. Think of Pregel and other iterative-style jobs. You may run several iterations before writing to disk to checkpoint. This can speed up your job immensely since you don't have to write & read data from slow disks. Of course you lose on fault tolerance but the speedup may be worth it.