7 ms·
Given that all the big data talk lately has been about GPU computing/TensorFlow, I'm glad to see that this Spark update shows in-memory computing is still viabl
by minimaxir 11y ago
Given that all the big data talk lately has been about GPU computing/TensorFlow, I'm glad to see that this Spark update shows in-memory computing is still viable. (Much cheaper to play with too!)
The key feature for me is Machine Learning functions in R, which otherwise lacks parallelizeable and scalable options. (Without resorting to black magic, anyways)
- luckydata 11y agoHonest question: what does "in memory" mean exactly? Do traditional databases not process data using memory? Doesn't Spark spill to disk when it reached the limit of the system memory? The in-memory thing always puzzled me.
- ihodes 11y agoI think that "in memory" is in contrast to traditional MR jobs that wrote to disk a lot more than Spark jobs do.
- cle 11y agoYou can "cache" a dataset in-memory and continue to perform operations on the dataset without having to persist it to disk, unlike MapReduce. Traditional databases can do a lot of that too, but aren't as scalable.
- btown 11y agoThe difference is that every variable is sharded by default. So if you have 100 machines with 8gb memory each, you can keep an 800gb array in memory. Some traditional databases can do this for one operation at a time, like querying a sharded index in Mongo or doing a range query in Cassandra. But usually you want to do a whole pipeline of these operations. In that case, your computation engine should own the memory for efficiency, and seamlessly redistribute based on access patterns. Thus, Spark and friends.
- abelo_the_cool 11y agohack tanki
- sgt101 11y agoPretty interesting to see watt for watt comparisons : ARM-64 with 64 cores vs GPUs. Also this is attractive to me because recursion.
- entee 11y agoNeural nets are fantastic, but not all problems either require or are best solved by neural nets. Sometimes you want to understand more deeply how certain variables contribute to the prediction. That's not as trivial to do with neural nets, but in a simple regression you can just look at the coefficients and have some idea of what matters most (assuming you've normalized everything properly). This is just one advantage of the more traditional models that Spark includes out of the box. Just because neural nets are awesome doesn't mean there aren't big advantages to the older stuff. Also, often it's not the algorithm that matters, it's how you prepare the data and how much data you have in hand. After all is said and done, in many cases the fancy model has similar performance to the boring model. Very happy to see Spark developing yet more abilities to make ML on large datasets seamless and (relatively) painless.
- Thaxll 11y agoMaybe there is a lot of talk about it but "everyone" is using Hadoop and the like.
- vonmoltke 11y agoEveryone who isn't still using grid computing for one reason or another, at least.
- deleted 11y ago[deleted]
- lmeyerov 11y agoThere are multiple (advanced) projects to get the next 10-100X of perf beyond this via in-memory & streaming GPU computing over columnar data. And... they're taking RDDs in as native formats, and some even target sql as the scripting lang :) FWIW, my interest here is what becomes possible for sub-second computations. We do interactive analytics, and GPUs become exciting for that, including in the distributed case :) Similar story for stuff like VR, real-time ML, etc.
- emcq 11y agoIt often makes sense to just use spark for event processing to create a shaker set of features that are used as input to more robust ML libraries like scikit learn.